[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:16 dist-test-slave-1vmg krb5kdc[29656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:16 dist-test-slave-1vmg krb5kdc[29656](info): set up 2 sockets
Aug 12 16:44:16 dist-test-slave-1vmg krb5kdc[29656](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 16:44:17.877113 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.581s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:18 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:18.115500 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36413
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:36413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:18.548724 29680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:18.549216 29680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:18.549368 29680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:18.557879 29680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:18.558051 29680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:18.558138 29680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:18.558319 29680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:18.558418 29680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:18.570174 29680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:36413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:18.572516 29680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:18.575021 29680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 16:44:18.599383 29680 server_base.cc:1034] running on GCE node
W20260812 16:44:18.599340 29688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:18.599730 29691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:18.599483 29687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:18.603099 29680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:18.606717 29680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:18.608237 29680 hybrid_clock.cc:648] HybridClock initialized: now 1786553058608121 us; error 161 us; skew 500 ppm
I20260812 16:44:18.609159 29680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:18 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553058, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:18.613991 29680 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:18.617132 29680 webserver.cc:460] Webserver started at http://127.28.243.62:38485/ using document root <none> and password file <none>
I20260812 16:44:18.618459 29680 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:18.618613 29680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:18.619218 29680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:18.625437 29680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0"
format_stamp: "Formatted at 2026-08-12 16:44:18 on dist-test-slave-1vmg"
server_key: "f3c1cee43b4e964aff62be775758395d"
server_key_iv: "0bd2b1abcc0a277201425127e5cdc195"
server_key_version: "encryptionkey@0"
I20260812 16:44:18.627415 29680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0"
format_stamp: "Formatted at 2026-08-12 16:44:18 on dist-test-slave-1vmg"
server_key: "f3c1cee43b4e964aff62be775758395d"
server_key_iv: "0bd2b1abcc0a277201425127e5cdc195"
server_key_version: "encryptionkey@0"
I20260812 16:44:18.639114 29680 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260812 16:44:18.645984 29699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:18.648399 29680 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:44:18.648723 29680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0"
format_stamp: "Formatted at 2026-08-12 16:44:18 on dist-test-slave-1vmg"
server_key: "f3c1cee43b4e964aff62be775758395d"
server_key_iv: "0bd2b1abcc0a277201425127e5cdc195"
server_key_version: "encryptionkey@0"
I20260812 16:44:18.648964 29680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:18.665669 29680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:18.673774 29680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:18.674188 29680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:18.702996 29680 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:36413
I20260812 16:44:18.703024 29789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:36413 every 8 connection(s)
I20260812 16:44:18.706674 29680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:18.714709 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:18.715050 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 29680
I20260812 16:44:18.715468 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:18.717746 29644 external_mini_cluster.cc:1428] Setting key d9ebe4ce1164bc60d548945d7d721377
I20260812 16:44:18.732885 29792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0: Bootstrap starting.
I20260812 16:44:18.739125 29792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:18.741612 29792 log.cc:824] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:18.746227 29792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0: No bootstrap required, opened a new log
Aug 12 16:44:18 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:18.757977 29792 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36413 } }
I20260812 16:44:18.758682 29792 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:18.758877 29792 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a3ae03b45e4c4e89aef18fe4d3a3c0, State: Initialized, Role: FOLLOWER
I20260812 16:44:18.760433 29792 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36413 } }
I20260812 16:44:18.760919 29792 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:18.761406 29792 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:18.761723 29792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:18.765707 29792 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36413 } }
I20260812 16:44:18.767309 29792 leader_election.cc:302] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a3ae03b45e4c4e89aef18fe4d3a3c0; no voters:
I20260812 16:44:18.768813 29792 leader_election.cc:288] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:18.769351 29800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:18.770589 29800 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [term 1 LEADER]: Becoming Leader. State: Replica: b2a3ae03b45e4c4e89aef18fe4d3a3c0, State: Running, Role: LEADER
I20260812 16:44:18.772190 29800 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36413 } }
I20260812 16:44:18.772457 29792 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:18.779313 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:18.725047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.28.243.62:36413)
0812 16:44:18.726362 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:18.726381 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:18.730401 (+ 4020us) server_negotiation.cc:407] Connection header received
0812 16:44:18.741930 (+ 11529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:18.741961 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:18.742785 (+ 824us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:18.743321 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:18.746887 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:18.749293 (+ 2406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:18.751159 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:18.752485 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:18.752783 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:18.760065 (+ 7282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:18.760152 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:18.760203 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:18.760265 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:18.766808 (+ 6543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:18.769422 (+ 2614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:18.769434 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:18.769448 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:18.769791 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:18.770642 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:18.770648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:18.770779 (+ 131us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:18.772480 (+ 1701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:18.772936 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:18.775087 (+ 2151us) server_negotiation.cc:299] Negotiation successful
0812 16:44:18.775730 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":443,"threads_started":1}
I20260812 16:44:18.779830 29802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2a3ae03b45e4c4e89aef18fe4d3a3c0. Latest consensus state: current_term: 1 leader_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36413 } } }
I20260812 16:44:18.780335 29802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:18.780797 29801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3ae03b45e4c4e89aef18fe4d3a3c0" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36413 } } }
I20260812 16:44:18.781189 29801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:18.791800 29812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:18.802716 29812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:18.821493 29812 catalog_manager.cc:1347] Generated new cluster ID: 2ca9bba433df49daa1b531fc0dc0022a
I20260812 16:44:18.821741 29812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:18.840127 29812 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:18.842379 29812 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:18.852859 29812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2a3ae03b45e4c4e89aef18fe4d3a3c0: Generated new TSK 0
I20260812 16:44:18.854323 29812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:18.943439 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36413
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:19.366060 29839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:19.366389 29839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:19.366591 29839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:19.375126 29839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:19.375273 29839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:19.375447 29839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:19.386708 29839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:19.388192 29839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:19.389670 29839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:19.405191 29846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:19.405622 29850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:19.405730 29854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:19.407048 29839 server_base.cc:1034] running on GCE node
I20260812 16:44:19.407805 29839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:19.408933 29839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:19.410207 29839 hybrid_clock.cc:648] HybridClock initialized: now 1786553059410145 us; error 49 us; skew 500 ppm
I20260812 16:44:19.410595 29839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:19 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553059, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:19.414400 29839 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:19.416142 29839 webserver.cc:460] Webserver started at http://127.28.243.1:43591/ using document root <none> and password file <none>
I20260812 16:44:19.416880 29839 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:19.417006 29839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:19.417315 29839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:19.420640 29839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "7843eb44d47c445f92d6f2b80e37e3f2"
format_stamp: "Formatted at 2026-08-12 16:44:19 on dist-test-slave-1vmg"
server_key: "09eea7b84ff6c3cb839cc4d8abc02495"
server_key_iv: "048f5ac98d8941e0f18c551a5f35cb1f"
server_key_version: "encryptionkey@0"
I20260812 16:44:19.421731 29839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "7843eb44d47c445f92d6f2b80e37e3f2"
format_stamp: "Formatted at 2026-08-12 16:44:19 on dist-test-slave-1vmg"
server_key: "09eea7b84ff6c3cb839cc4d8abc02495"
server_key_iv: "048f5ac98d8941e0f18c551a5f35cb1f"
server_key_version: "encryptionkey@0"
I20260812 16:44:19.428310 29839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:44:19.433344 29868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:19.435114 29839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:44:19.435405 29839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "7843eb44d47c445f92d6f2b80e37e3f2"
format_stamp: "Formatted at 2026-08-12 16:44:19 on dist-test-slave-1vmg"
server_key: "09eea7b84ff6c3cb839cc4d8abc02495"
server_key_iv: "048f5ac98d8941e0f18c551a5f35cb1f"
server_key_version: "encryptionkey@0"
I20260812 16:44:19.435632 29839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:19.451874 29839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:19.456961 29839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:19.457288 29839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:19.458603 29839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:19.460458 29839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:19.460543 29839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:19.460727 29839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:19.460832 29839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:19.503389 29839 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:44535
I20260812 16:44:19.503437 30054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:44535 every 8 connection(s)
I20260812 16:44:19.505084 29839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:19.509268 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 29839
I20260812 16:44:19.509507 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:19.510385 29644 external_mini_cluster.cc:1428] Setting key 23c48d9265dce9e1a9b6eef281ea0ebf
Aug 12 16:44:19 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553059, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:44:19.530306 30057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:19.509072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:57085 (local address 127.28.243.62:36413)
0812 16:44:19.509562 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:19.509567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:19.510887 (+ 1320us) server_negotiation.cc:407] Connection header received
0812 16:44:19.513379 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:19.513385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:19.513476 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:19.513580 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:19.515902 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:19.516681 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:19.518721 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:19.519220 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:19.519345 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:19.523536 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:19.523542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:19.523545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:19.523587 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:19.526570 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:19.527592 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:19.527597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:19.527599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:19.527736 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:19.528215 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:19.528219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:19.528221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:19.528526 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:19.528669 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:19.529552 (+ 883us) server_negotiation.cc:299] Negotiation successful
0812 16:44:19.529824 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
I20260812 16:44:19.531435 30058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:19.509802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36413 (local address 127.28.243.1:57085)
0812 16:44:19.510704 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:19.510761 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:19.513017 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:19.513877 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:19.513887 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:19.514649 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:19.515596 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:19.515618 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:19.516867 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:19.516872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:19.518516 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:19.518533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:19.519486 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:19.519495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:19.519753 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:19.520511 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:19.520537 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:19.523235 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:19.526834 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:19.526845 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:19.526863 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:19.527439 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:19.527864 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:19.527868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:19.527870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:19.528086 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:19.528662 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:19.528671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:19.529069 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 16:44:19.529405 (+ 336us) client_negotiation.cc:240] Negotiation successful
0812 16:44:19.530011 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":380,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
I20260812 16:44:19.533409 30055 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36413
I20260812 16:44:19.534194 30055 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:19.535351 30055 heartbeater.cc:511] Master 127.28.243.62:36413 requested a full tablet report, sending...
I20260812 16:44:19.542089 29729 ts_manager.cc:194] Registered new tserver with Master: 7843eb44d47c445f92d6f2b80e37e3f2 (127.28.243.1:44535)
I20260812 16:44:19.544842 29729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:57085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:19.564450 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36413
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:19.968451 30061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:19.968807 30061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:19.968920 30061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:19.977412 30061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:19.977566 30061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:19.977689 30061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:19.989328 30061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:19.990803 30061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:19.992875 30061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:20.009794 30072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:20.009781 30077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:20.009831 30073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:20.010823 30061 server_base.cc:1034] running on GCE node
I20260812 16:44:20.011639 30061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:20.012766 30061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:20.014037 30061 hybrid_clock.cc:648] HybridClock initialized: now 1786553060013985 us; error 40 us; skew 500 ppm
I20260812 16:44:20.014433 30061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:20.019163 30061 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:20.021380 30061 webserver.cc:460] Webserver started at http://127.28.243.2:33531/ using document root <none> and password file <none>
I20260812 16:44:20.022194 30061 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:20.022353 30061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:20.022835 30061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:20.026835 30061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "6ba06fe44ee040a29d385c24dafe7db5"
format_stamp: "Formatted at 2026-08-12 16:44:20 on dist-test-slave-1vmg"
server_key: "29c5e0280fdb61611ad832494c4c187f"
server_key_iv: "c0c14d41c963926fa99a7d9e78996ba9"
server_key_version: "encryptionkey@0"
I20260812 16:44:20.029808 30061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "6ba06fe44ee040a29d385c24dafe7db5"
format_stamp: "Formatted at 2026-08-12 16:44:20 on dist-test-slave-1vmg"
server_key: "29c5e0280fdb61611ad832494c4c187f"
server_key_iv: "c0c14d41c963926fa99a7d9e78996ba9"
server_key_version: "encryptionkey@0"
I20260812 16:44:20.037410 30061 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 16:44:20.042124 30088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:20.044625 30061 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 16:44:20.044849 30061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "6ba06fe44ee040a29d385c24dafe7db5"
format_stamp: "Formatted at 2026-08-12 16:44:20 on dist-test-slave-1vmg"
server_key: "29c5e0280fdb61611ad832494c4c187f"
server_key_iv: "c0c14d41c963926fa99a7d9e78996ba9"
server_key_version: "encryptionkey@0"
I20260812 16:44:20.045536 30061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:20.073127 30061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:20.079068 30061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:20.079501 30061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:20.081060 30061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:20.084157 30061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:20.084358 30061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:20.084542 30061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:20.084604 30061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:20.137327 30061 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:33645
I20260812 16:44:20.137391 30274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:33645 every 8 connection(s)
I20260812 16:44:20.138819 30061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:20.140362 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30061
I20260812 16:44:20.140638 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:20.141194 29644 external_mini_cluster.cc:1428] Setting key 03efca0225f14b4b30f2186366663255
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
I20260812 16:44:20.168501 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.142559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:45281 (local address 127.28.243.62:36413)
0812 16:44:20.143089 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:20.143095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:20.144697 (+ 1602us) server_negotiation.cc:407] Connection header received
0812 16:44:20.147371 (+ 2674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:20.147378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:20.147468 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:20.147601 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:20.151728 (+ 4127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.152786 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.155148 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.155913 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.156056 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.160751 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:20.160760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:20.160764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:20.160818 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:20.164102 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.165314 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.165319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.165321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.165483 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.166001 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.166006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.166010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.166353 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:20.166483 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:20.168007 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0812 16:44:20.168228 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":240,"threads_started":1}
I20260812 16:44:20.170430 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.143401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36413 (local address 127.28.243.2:45281)
0812 16:44:20.144487 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:20.144546 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:20.147050 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:20.148324 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:20.148335 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:20.149852 (+ 1517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:20.151321 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:20.151458 (+ 137us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.153059 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.153064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:20.154940 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:20.154967 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.156276 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.156284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:20.156619 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.157486 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:20.157515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:20.160348 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:20.164311 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:20.164321 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:20.164339 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:20.165136 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:20.165594 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:20.165598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:20.165600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:20.165862 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:20.166564 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:20.166578 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:20.167495 (+ 917us) client_negotiation.cc:769] Sending connection context
0812 16:44:20.167885 (+ 390us) client_negotiation.cc:240] Negotiation successful
0812 16:44:20.168642 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":522,"threads_started":1}
I20260812 16:44:20.172444 30275 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36413
I20260812 16:44:20.172858 30275 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:20.173811 30275 heartbeater.cc:511] Master 127.28.243.62:36413 requested a full tablet report, sending...
I20260812 16:44:20.176187 29729 ts_manager.cc:194] Registered new tserver with Master: 6ba06fe44ee040a29d385c24dafe7db5 (127.28.243.2:33645)
I20260812 16:44:20.177107 29729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:45281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:20.198256 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36413
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:20.549094 30055 heartbeater.cc:503] Master 127.28.243.62:36413 was elected leader, sending a full tablet report...
W20260812 16:44:20.610365 30282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:20.610728 30282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:20.610949 30282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:20.621441 30282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:20.621613 30282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:20.621780 30282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:20.634656 30282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:20.636211 30282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:20.637717 30282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:20.657721 30290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:20.658164 30295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:20.658354 30289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:20.658799 30282 server_base.cc:1034] running on GCE node
I20260812 16:44:20.659626 30282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:20.660771 30282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:20.662063 30282 hybrid_clock.cc:648] HybridClock initialized: now 1786553060661996 us; error 43 us; skew 500 ppm
I20260812 16:44:20.662503 30282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:20.668409 30282 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:20.670760 30282 webserver.cc:460] Webserver started at http://127.28.243.3:34449/ using document root <none> and password file <none>
I20260812 16:44:20.671675 30282 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:20.671808 30282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:20.672179 30282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:20.675880 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "323f962e0a0b4b74a4aff749d2552ecb"
format_stamp: "Formatted at 2026-08-12 16:44:20 on dist-test-slave-1vmg"
server_key: "42c34a47d97897d3f37978aac9cff9ea"
server_key_iv: "b699cd8fd8d86a9d68de984b8e20a62c"
server_key_version: "encryptionkey@0"
I20260812 16:44:20.677006 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "323f962e0a0b4b74a4aff749d2552ecb"
format_stamp: "Formatted at 2026-08-12 16:44:20 on dist-test-slave-1vmg"
server_key: "42c34a47d97897d3f37978aac9cff9ea"
server_key_iv: "b699cd8fd8d86a9d68de984b8e20a62c"
server_key_version: "encryptionkey@0"
I20260812 16:44:20.684458 30282 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 16:44:20.690465 30309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:20.692443 30282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:44:20.692633 30282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "323f962e0a0b4b74a4aff749d2552ecb"
format_stamp: "Formatted at 2026-08-12 16:44:20 on dist-test-slave-1vmg"
server_key: "42c34a47d97897d3f37978aac9cff9ea"
server_key_iv: "b699cd8fd8d86a9d68de984b8e20a62c"
server_key_version: "encryptionkey@0"
I20260812 16:44:20.692890 30282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:20.717005 30282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:20.722720 30282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:20.724045 30282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:20.726383 30282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:20.729719 30282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:20.729911 30282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:20.730049 30282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:20.730103 30282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:20.776870 30282 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:41641
I20260812 16:44:20.777046 30491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:41641 every 8 connection(s)
I20260812 16:44:20.778456 30282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:20.785334 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30282
I20260812 16:44:20.785538 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:20.786130 29644 external_mini_cluster.cc:1428] Setting key 68e9606df352bdf9d9535280e3e5d3c0
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:20.807091 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.783985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:59933 (local address 127.28.243.62:36413)
0812 16:44:20.784576 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:20.784581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:20.787071 (+ 2490us) server_negotiation.cc:407] Connection header received
0812 16:44:20.789209 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:20.789214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:20.789306 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:20.789439 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:20.791750 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.792672 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.794380 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.794814 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.794955 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.798948 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:20.798956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:20.798961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:20.799004 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:20.801908 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.803145 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.803150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.803152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.803327 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.803949 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.803955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.803958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.805273 (+ 1315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:20.805427 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:20.806491 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0812 16:44:20.806706 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":316,"threads_started":1}
I20260812 16:44:20.808238 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.785534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36413 (local address 127.28.243.3:59933)
0812 16:44:20.786809 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:20.786876 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:20.788917 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:20.789813 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:20.789823 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:20.790775 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:20.791527 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:20.791551 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.792857 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.792862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:20.794214 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:20.794231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.795106 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.795112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:20.795338 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.796061 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:20.796094 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:20.798650 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:20.802134 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:20.802143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:20.802160 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:20.802997 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:20.803453 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:20.803460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:20.803464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:20.803762 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:20.805439 (+ 1677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:20.805448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:20.805905 (+ 457us) client_negotiation.cc:769] Sending connection context
0812 16:44:20.806340 (+ 435us) client_negotiation.cc:240] Negotiation successful
0812 16:44:20.806799 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":573,"threads_started":1}
I20260812 16:44:20.810134 30492 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36413
I20260812 16:44:20.810472 30492 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:20.811354 30492 heartbeater.cc:511] Master 127.28.243.62:36413 requested a full tablet report, sending...
I20260812 16:44:20.814210 29729 ts_manager.cc:194] Registered new tserver with Master: 323f962e0a0b4b74a4aff749d2552ecb (127.28.243.3:41641)
I20260812 16:44:20.815181 29729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:59933
I20260812 16:44:20.818653 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:20.880275 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.859193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.28.243.1:44535)
0812 16:44:20.859929 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:20.859937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:20.859970 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:44:20.860186 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:20.860194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:20.860436 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:20.860578 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:20.863216 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.864922 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.867059 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.867760 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.867901 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.872987 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:20.873007 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:20.873025 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:20.873079 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:20.876628 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.877588 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.877597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.877599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.877805 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.878393 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.878398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.878400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.878856 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:20.879180 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:20.879567 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 16:44:20.879837 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":370,"threads_started":1}
W20260812 16:44:20.881937 29913 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38964
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:44:20 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:20.937152 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.920334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.28.243.1:44535)
0812 16:44:20.920608 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:20.920615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:20.920762 (+ 147us) server_negotiation.cc:407] Connection header received
0812 16:44:20.920954 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:20.920959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:20.921029 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:20.921116 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:20.922774 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.923519 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.924845 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.925310 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.925428 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.930336 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:20.930343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:20.930346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:20.930389 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:20.933653 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.935041 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.935047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.935051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.935295 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.935707 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.935711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.935713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.935993 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:20.936236 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:20.936578 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 16:44:20.936817 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 16:44:20.939208 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 29839
I20260812 16:44:20.955091 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:20.940441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.28.243.1:44535)
0812 16:44:20.940618 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:20.940623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:20.941039 (+ 416us) server_negotiation.cc:407] Connection header received
0812 16:44:20.941163 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:20.941168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:20.941273 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:20.941472 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:20.942820 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.943629 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.945067 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:20.945501 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:20.945640 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:20.948198 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:20.948205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:20.948208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:20.948247 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:20.951208 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.951998 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.952002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.952004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.952231 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:20.952715 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:20.952719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:20.952722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:20.953354 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:20.953607 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:20.954365 (+ 758us) server_negotiation.cc:299] Negotiation successful
0812 16:44:20.954740 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 16:44:20.956195 30050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:21.049754 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 29839
I20260812 16:44:21.075377 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30061
Aug 12 16:44:21 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:44:21.094396 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:21.076484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.28.243.2:33645)
0812 16:44:21.077458 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:21.077470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:21.077513 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:44:21.077612 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:21.077621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:21.078068 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:21.078305 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:21.079703 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:21.080839 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:21.082333 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:21.082963 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:21.083100 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:21.087554 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:21.087571 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:21.087588 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:21.087671 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:21.091571 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:21.092381 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:21.092389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:21.092392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:21.092537 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:21.092906 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:21.092909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:21.092911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:21.093290 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:21.093541 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:21.093861 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 16:44:21.094089 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":540,"threads_started":1}
I20260812 16:44:21.095966 30265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:21.186317 30275 heartbeater.cc:503] Master 127.28.243.62:36413 was elected leader, sending a full tablet report...
I20260812 16:44:21.187820 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30061
I20260812 16:44:21.214296 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30282
W20260812 16:44:21.214145 29703 connection.cc:535] server connection from 127.28.243.2:45281 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Aug 12 16:44:21 dist-test-slave-1vmg krb5kdc[29656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:44:21.233646 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:21.215377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.28.243.3:41641)
0812 16:44:21.215985 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:21.215992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:21.216024 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:44:21.216086 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:21.216092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:21.216337 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:21.216555 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:21.217807 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:21.218860 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:21.220251 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:21.220852 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:21.220965 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:21.225836 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:21.225855 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:21.225873 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:21.225926 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:21.229412 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:21.230482 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:21.230505 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:21.230508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:21.230784 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:21.231474 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:21.231479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:21.231481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:21.232064 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:21.232422 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:21.232930 (+ 508us) server_negotiation.cc:299] Negotiation successful
0812 16:44:21.233283 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":324,"threads_started":1}
I20260812 16:44:21.235280 30484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:21.320331 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30282
I20260812 16:44:21.348683 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 29680
I20260812 16:44:21.349857 29781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:21.455650 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 29680
2026-08-12T16:44:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5235 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:21 dist-test-slave-1vmg krb5kdc[30526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:21 dist-test-slave-1vmg krb5kdc[30526](info): set up 2 sockets
Aug 12 16:44:21 dist-test-slave-1vmg krb5kdc[30526](info): commencing operation
krb5kdc: starting...
W20260812 16:44:22.747651 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:22 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:22.979597 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42257
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:23.391886 30553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:23.392241 30553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:23.392387 30553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:23.400863 30553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:23.401013 30553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:23.401062 30553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:23.401141 30553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:23.401206 30553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:23.413823 30553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:23.415338 30553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:23.416985 30553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:23.431998 30561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:23.431950 30566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:23.431967 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:23.433018 30553 server_base.cc:1034] running on GCE node
I20260812 16:44:23.433929 30553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:23.435592 30553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:23.436846 30553 hybrid_clock.cc:648] HybridClock initialized: now 1786553063436794 us; error 40 us; skew 500 ppm
I20260812 16:44:23.437345 30553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:23 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:23.440838 30553 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:23.442555 30553 webserver.cc:460] Webserver started at http://127.28.243.62:43875/ using document root <none> and password file <none>
I20260812 16:44:23.443408 30553 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:23.443542 30553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:23.443872 30553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:23.447135 30553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "64d206df68ad42be8cc791e7e39ba76e"
format_stamp: "Formatted at 2026-08-12 16:44:23 on dist-test-slave-1vmg"
server_key: "560de3559b8847e49fbb0575ab93986f"
server_key_iv: "7b2e80ac3400ea6ba81be190d46b063e"
server_key_version: "encryptionkey@0"
I20260812 16:44:23.448185 30553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "64d206df68ad42be8cc791e7e39ba76e"
format_stamp: "Formatted at 2026-08-12 16:44:23 on dist-test-slave-1vmg"
server_key: "560de3559b8847e49fbb0575ab93986f"
server_key_iv: "7b2e80ac3400ea6ba81be190d46b063e"
server_key_version: "encryptionkey@0"
I20260812 16:44:23.456140 30553 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:44:23.460443 30576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:23.462251 30553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:44:23.462466 30553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "64d206df68ad42be8cc791e7e39ba76e"
format_stamp: "Formatted at 2026-08-12 16:44:23 on dist-test-slave-1vmg"
server_key: "560de3559b8847e49fbb0575ab93986f"
server_key_iv: "7b2e80ac3400ea6ba81be190d46b063e"
server_key_version: "encryptionkey@0"
I20260812 16:44:23.462692 30553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:23.484033 30553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:23.489959 30553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:23.490330 30553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:23.514477 30553 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:42257
I20260812 16:44:23.514479 30650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:42257 every 8 connection(s)
I20260812 16:44:23.516525 30553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:23.522181 30653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:23.526649 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30553
I20260812 16:44:23.526953 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:23.527868 29644 external_mini_cluster.cc:1428] Setting key 7c27c97fb1a26dceb5912f5f81b9b245
I20260812 16:44:23.536736 30653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e: Bootstrap starting.
I20260812 16:44:23.540798 30653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:23.542315 30653 log.cc:824] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:44:23 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:23.547314 30653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e: No bootstrap required, opened a new log
I20260812 16:44:23.559362 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:23.530493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.28.243.62:42257)
0812 16:44:23.531412 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:23.531428 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:23.531491 (+ 63us) server_negotiation.cc:407] Connection header received
0812 16:44:23.534687 (+ 3196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:23.534725 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:23.535368 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:23.535892 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:23.537622 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:23.539721 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:23.541180 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:23.542273 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:23.542482 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:23.547897 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:23.547911 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:23.547935 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:23.548013 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:23.553499 (+ 5486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:23.554514 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:23.554525 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:23.554535 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:23.554818 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:23.555372 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:23.555377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:23.555381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:23.556278 (+ 897us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:23.556614 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:23.557190 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 16:44:23.557766 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":368,"threads_started":1}
I20260812 16:44:23.559754 30653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d206df68ad42be8cc791e7e39ba76e" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42257 } }
I20260812 16:44:23.560187 30653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:23.560295 30653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d206df68ad42be8cc791e7e39ba76e, State: Initialized, Role: FOLLOWER
I20260812 16:44:23.560920 30653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d206df68ad42be8cc791e7e39ba76e" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42257 } }
I20260812 16:44:23.561205 30653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:23.561372 30653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:23.562089 30653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:23.565719 30653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d206df68ad42be8cc791e7e39ba76e" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42257 } }
I20260812 16:44:23.566329 30653 leader_election.cc:302] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d206df68ad42be8cc791e7e39ba76e; no voters:
I20260812 16:44:23.567078 30653 leader_election.cc:288] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:23.567361 30664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:23.568791 30664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [term 1 LEADER]: Becoming Leader. State: Replica: 64d206df68ad42be8cc791e7e39ba76e, State: Running, Role: LEADER
I20260812 16:44:23.569568 30664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d206df68ad42be8cc791e7e39ba76e" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42257 } }
I20260812 16:44:23.569921 30653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:23.573855 30668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d206df68ad42be8cc791e7e39ba76e. Latest consensus state: current_term: 1 leader_uuid: "64d206df68ad42be8cc791e7e39ba76e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d206df68ad42be8cc791e7e39ba76e" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42257 } } }
I20260812 16:44:23.573830 30667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d206df68ad42be8cc791e7e39ba76e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d206df68ad42be8cc791e7e39ba76e" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42257 } } }
I20260812 16:44:23.574434 30667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:23.574434 30668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:23.578303 30675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:23.583417 30675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:23.594588 30675 catalog_manager.cc:1347] Generated new cluster ID: 1f38d1ecd54b4654958bd6adcc6a1264
I20260812 16:44:23.594764 30675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:23.607532 30675 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:23.609977 30675 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:23.619992 30675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d206df68ad42be8cc791e7e39ba76e: Generated new TSK 0
I20260812 16:44:23.621141 30675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:23.683410 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42257
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:24.088560 30694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:24.088886 30694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:24.089008 30694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:24.096553 30694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:24.096693 30694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:24.096855 30694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:24.107290 30694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:24.108800 30694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:24.110319 30694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:24.126616 30701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:24.126654 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:24.126562 30702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:24.127840 30694 server_base.cc:1034] running on GCE node
I20260812 16:44:24.128749 30694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:24.130637 30694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:24.131959 30694 hybrid_clock.cc:648] HybridClock initialized: now 1786553064131858 us; error 96 us; skew 500 ppm
I20260812 16:44:24.132359 30694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:24 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553064, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:24.136312 30694 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:24.138139 30694 webserver.cc:460] Webserver started at http://127.28.243.1:33145/ using document root <none> and password file <none>
I20260812 16:44:24.139024 30694 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:24.139153 30694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:24.139492 30694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:24.142761 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "54779cbb495f476893a7e6d408321b70"
format_stamp: "Formatted at 2026-08-12 16:44:24 on dist-test-slave-1vmg"
server_key: "71fb9fee645e6ebc4bb878a101f270f0"
server_key_iv: "9b5dc203d658f8c3bfc8519f60fc58fa"
server_key_version: "encryptionkey@0"
I20260812 16:44:24.143759 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "54779cbb495f476893a7e6d408321b70"
format_stamp: "Formatted at 2026-08-12 16:44:24 on dist-test-slave-1vmg"
server_key: "71fb9fee645e6ebc4bb878a101f270f0"
server_key_iv: "9b5dc203d658f8c3bfc8519f60fc58fa"
server_key_version: "encryptionkey@0"
I20260812 16:44:24.151154 30694 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:44:24.155643 30713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:24.157812 30694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:44:24.158035 30694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "54779cbb495f476893a7e6d408321b70"
format_stamp: "Formatted at 2026-08-12 16:44:24 on dist-test-slave-1vmg"
server_key: "71fb9fee645e6ebc4bb878a101f270f0"
server_key_iv: "9b5dc203d658f8c3bfc8519f60fc58fa"
server_key_version: "encryptionkey@0"
I20260812 16:44:24.158285 30694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:24.174616 30694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:24.181071 30694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:24.181742 30694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:24.183107 30694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:24.185107 30694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:24.185206 30694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:24.185322 30694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:24.185391 30694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:24.221576 30694 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:39697
I20260812 16:44:24.221616 30898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:39697 every 8 connection(s)
I20260812 16:44:24.223074 30694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:24.230540 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30694
I20260812 16:44:24.230789 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:24.231462 29644 external_mini_cluster.cc:1428] Setting key 5bd1b5c44e7444966192528b2bd85ada
Aug 12 16:44:24 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553064, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:24.249446 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:24.226026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:51107 (local address 127.28.243.62:42257)
0812 16:44:24.226563 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:24.226570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:24.227737 (+ 1167us) server_negotiation.cc:407] Connection header received
0812 16:44:24.229942 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:24.229949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:24.230070 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:24.230307 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:24.233043 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.234008 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.236009 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.236445 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.236581 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:24.241421 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:24.241429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:24.241432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:24.241481 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:24.244429 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:24.245712 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:24.245717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:24.245720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:24.245883 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:24.246622 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:24.246627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:24.246629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:24.247018 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:24.247174 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:24.248907 (+ 1733us) server_negotiation.cc:299] Negotiation successful
0812 16:44:24.249157 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":248,"threads_started":1}
I20260812 16:44:24.251346 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:24.226616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42257 (local address 127.28.243.1:51107)
0812 16:44:24.227532 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:24.227585 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:24.229390 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:24.230843 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:24.230863 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:24.231786 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:24.232803 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:24.232836 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.234219 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.234224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:24.235826 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:24.235845 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.236744 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.236753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:24.237038 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:24.238089 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:24.238137 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:24.241023 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:24.244692 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:24.244707 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:24.244730 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:24.245491 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:24.246029 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:24.246036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:24.246040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:24.246411 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:24.247226 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:24.247241 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:24.248020 (+ 779us) client_negotiation.cc:769] Sending connection context
0812 16:44:24.248513 (+ 493us) client_negotiation.cc:240] Negotiation successful
0812 16:44:24.249352 (+ 839us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":413,"threads_started":1}
I20260812 16:44:24.253356 30900 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42257
I20260812 16:44:24.253774 30900 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:44:24.254899 30900 heartbeater.cc:511] Master 127.28.243.62:42257 requested a full tablet report, sending...
I20260812 16:44:24.257565 30600 ts_manager.cc:194] Registered new tserver with Master: 54779cbb495f476893a7e6d408321b70 (127.28.243.1:39697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
I20260812 16:44:24.259234 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:51107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:24.287745 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42257
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:24.702061 30912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:24.702580 30912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:24.702711 30912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:24.711501 30912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:24.711696 30912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:24.711903 30912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:24.724115 30912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:24.726183 30912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:24.728129 30912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:24.747380 30922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:24.747324 30926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:24.747391 30921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:24.748652 30912 server_base.cc:1034] running on GCE node
I20260812 16:44:24.749634 30912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:24.751076 30912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:24.752405 30912 hybrid_clock.cc:648] HybridClock initialized: now 1786553064752322 us; error 52 us; skew 500 ppm
I20260812 16:44:24.752907 30912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:24 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553064, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:24.757182 30912 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:24.759112 30912 webserver.cc:460] Webserver started at http://127.28.243.2:41555/ using document root <none> and password file <none>
I20260812 16:44:24.759955 30912 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:24.760107 30912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:24.760457 30912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:24.764047 30912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "b7701001bace4615ac9700ec49276072"
format_stamp: "Formatted at 2026-08-12 16:44:24 on dist-test-slave-1vmg"
server_key: "3e5513b6f72e3cfb59aa13855aaa54f7"
server_key_iv: "4d598320e3f56de89cba5aa148b2f30f"
server_key_version: "encryptionkey@0"
I20260812 16:44:24.764948 30912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "b7701001bace4615ac9700ec49276072"
format_stamp: "Formatted at 2026-08-12 16:44:24 on dist-test-slave-1vmg"
server_key: "3e5513b6f72e3cfb59aa13855aaa54f7"
server_key_iv: "4d598320e3f56de89cba5aa148b2f30f"
server_key_version: "encryptionkey@0"
I20260812 16:44:24.771926 30912 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260812 16:44:24.776798 30938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:24.778590 30912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:44:24.778824 30912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "b7701001bace4615ac9700ec49276072"
format_stamp: "Formatted at 2026-08-12 16:44:24 on dist-test-slave-1vmg"
server_key: "3e5513b6f72e3cfb59aa13855aaa54f7"
server_key_iv: "4d598320e3f56de89cba5aa148b2f30f"
server_key_version: "encryptionkey@0"
I20260812 16:44:24.779103 30912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:24.797060 30912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:24.802522 30912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:24.802847 30912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:24.804203 30912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:24.805882 30912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:24.805979 30912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:24.806115 30912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:24.806180 30912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:24.842808 30912 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:41009
I20260812 16:44:24.842841 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:41009 every 8 connection(s)
I20260812 16:44:24.844306 30912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:24.846261 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30912
I20260812 16:44:24.846673 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:24.847353 29644 external_mini_cluster.cc:1428] Setting key 147f399cdd0416d1738039af70807edd
Aug 12 16:44:24 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553064, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:24.872785 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:24.848512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:33029 (local address 127.28.243.62:42257)
0812 16:44:24.849208 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:24.849214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:24.853778 (+ 4564us) server_negotiation.cc:407] Connection header received
0812 16:44:24.853881 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:24.853888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:24.854001 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:24.854158 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:24.856248 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.857390 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.859319 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.860068 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.860241 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:24.865277 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:24.865284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:24.865288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:24.865331 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:24.868346 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:24.869658 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:24.869662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:24.869664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:24.869805 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:24.870437 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:24.870441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:24.870443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:24.870806 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:24.870972 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:24.872266 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0812 16:44:24.872480 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
I20260812 16:44:24.874303 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:24.849022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42257 (local address 127.28.243.2:33029)
0812 16:44:24.849998 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:24.850044 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:24.853233 (+ 3189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:24.854495 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:24.854507 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:24.855232 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:24.855991 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:24.856014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.857644 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.857649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:24.859123 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:24.859140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:24.860444 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:24.860450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:24.860698 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:24.861479 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:24.861505 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:24.864821 (+ 3316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:24.868618 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:24.868632 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:24.868656 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:24.869496 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:24.869924 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:24.869929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:24.869932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:24.870241 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:24.871091 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:24.871101 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:24.871756 (+ 655us) client_negotiation.cc:769] Sending connection context
0812 16:44:24.872147 (+ 391us) client_negotiation.cc:240] Negotiation successful
0812 16:44:24.872807 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":381,"threads_started":1}
I20260812 16:44:24.876372 31121 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
I20260812 16:44:24.876745 31121 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:24.877576 31121 heartbeater.cc:511] Master 127.28.243.62:42257 requested a full tablet report, sending...
I20260812 16:44:24.879470 30600 ts_manager.cc:194] Registered new tserver with Master: b7701001bace4615ac9700ec49276072 (127.28.243.2:41009)
I20260812 16:44:24.880587 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:33029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:24.907508 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42257
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:25.262727 30900 heartbeater.cc:503] Master 127.28.243.62:42257 was elected leader, sending a full tablet report...
W20260812 16:44:25.315904 31135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:25.316243 31135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:25.316380 31135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:25.324615 31135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:25.324780 31135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:25.324940 31135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:25.340628 31135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:25.342166 31135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:25.343729 31135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:25.359647 31143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:25.359548 31144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:25.359572 31148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:25.360534 31135 server_base.cc:1034] running on GCE node
I20260812 16:44:25.361259 31135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:25.362418 31135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:25.363655 31135 hybrid_clock.cc:648] HybridClock initialized: now 1786553065363592 us; error 59 us; skew 500 ppm
I20260812 16:44:25.364060 31135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:25 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:25.368381 31135 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:25.371230 31135 webserver.cc:460] Webserver started at http://127.28.243.3:35925/ using document root <none> and password file <none>
I20260812 16:44:25.372376 31135 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:25.372522 31135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:25.372951 31135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:25.377487 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "e055262d02e24f2c97cb393b30c75bbf"
format_stamp: "Formatted at 2026-08-12 16:44:25 on dist-test-slave-1vmg"
server_key: "9d4236514f6e46af747931f7636e7abc"
server_key_iv: "43256fad8c49b0f3bed3621ed778b115"
server_key_version: "encryptionkey@0"
I20260812 16:44:25.378499 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "e055262d02e24f2c97cb393b30c75bbf"
format_stamp: "Formatted at 2026-08-12 16:44:25 on dist-test-slave-1vmg"
server_key: "9d4236514f6e46af747931f7636e7abc"
server_key_iv: "43256fad8c49b0f3bed3621ed778b115"
server_key_version: "encryptionkey@0"
I20260812 16:44:25.386874 31135 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260812 16:44:25.392271 31161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:25.394021 31135 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260812 16:44:25.394228 31135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "e055262d02e24f2c97cb393b30c75bbf"
format_stamp: "Formatted at 2026-08-12 16:44:25 on dist-test-slave-1vmg"
server_key: "9d4236514f6e46af747931f7636e7abc"
server_key_iv: "43256fad8c49b0f3bed3621ed778b115"
server_key_version: "encryptionkey@0"
I20260812 16:44:25.394435 31135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:25.428117 31135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:25.433562 31135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:25.433898 31135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:25.435039 31135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:25.436633 31135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:25.436717 31135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:25.436863 31135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:25.436920 31135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:25.479800 31135 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:33077
I20260812 16:44:25.479871 31368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:33077 every 8 connection(s)
I20260812 16:44:25.481455 31135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:25.484517 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31135
I20260812 16:44:25.484781 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:25.485301 29644 external_mini_cluster.cc:1428] Setting key b7681c7b65446c855e531bdd49445096
Aug 12 16:44:25 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:25.509433 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.484731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:49353 (local address 127.28.243.62:42257)
0812 16:44:25.485267 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.485272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.487424 (+ 2152us) server_negotiation.cc:407] Connection header received
0812 16:44:25.491307 (+ 3883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.491313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.491416 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.491556 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:25.493334 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.494151 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.496695 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.497283 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.497458 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.502385 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:25.502394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:25.502398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:25.502450 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:25.505589 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:25.507064 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:25.507068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:25.507070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:25.507217 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:25.507672 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:25.507675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:25.507677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:25.507969 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:25.508096 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:25.508930 (+ 834us) server_negotiation.cc:299] Negotiation successful
0812 16:44:25.509129 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":245,"threads_started":1}
I20260812 16:44:25.511758 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.485512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42257 (local address 127.28.243.3:49353)
0812 16:44:25.487128 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:25.487221 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:25.490888 (+ 3667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:25.491803 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:25.491813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:25.492441 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:25.493116 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.493139 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.494477 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.494492 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.496410 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.496432 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.497720 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.497728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.498346 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.499257 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:25.499288 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:25.502026 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:25.506058 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:25.506082 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:25.506101 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:25.506841 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:25.507314 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:25.507317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:25.507319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:25.507541 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:25.508091 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:25.508099 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:25.508475 (+ 376us) client_negotiation.cc:769] Sending connection context
0812 16:44:25.508759 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 16:44:25.509206 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":516,"threads_started":1}
I20260812 16:44:25.513437 31370 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42257
I20260812 16:44:25.513942 31370 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:25.515096 31370 heartbeater.cc:511] Master 127.28.243.62:42257 requested a full tablet report, sending...
I20260812 16:44:25.517230 30600 ts_manager.cc:194] Registered new tserver with Master: e055262d02e24f2c97cb393b30c75bbf (127.28.243.3:33077)
I20260812 16:44:25.518236 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:49353
I20260812 16:44:25.530624 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:44:25.555979 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.540760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.28.243.62:42257)
0812 16:44:25.541050 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.541057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.541686 (+ 629us) server_negotiation.cc:407] Connection header received
0812 16:44:25.541876 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.541882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.541968 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.542074 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:25.543828 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.544631 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.546037 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.546489 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.546637 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.549326 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:25.549332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:25.549335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:25.549375 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:25.552691 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:25.553662 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:25.553668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:25.553670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:25.553836 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:25.554246 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:25.554251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:25.554253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:25.554610 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:25.554876 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:25.555355 (+ 479us) server_negotiation.cc:299] Negotiation successful
0812 16:44:25.555622 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 16:44:25.567255 30600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:44:25.578704 30600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:25.616106 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.600739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:33077 (local address 127.0.0.1:40694)
0812 16:44:25.602180 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:25.602236 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:25.602500 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:25.603976 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:25.603986 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:25.604025 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:25.604827 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.604844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.608332 (+ 3488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.608340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.611537 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.611567 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.614742 (+ 3175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.614751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.615095 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.615165 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 16:44:25.615348 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 16:44:25.615578 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":898,"threads_started":1}
I20260812 16:44:25.617399 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.600873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.28.243.3:33077)
0812 16:44:25.601753 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.601762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.602710 (+ 948us) server_negotiation.cc:407] Connection header received
0812 16:44:25.602818 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.602826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.603381 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.603621 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:25.605558 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.607910 (+ 2352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.611884 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.613895 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.614068 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.614441 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:25.616706 (+ 2265us) server_negotiation.cc:299] Negotiation successful
0812 16:44:25.616965 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":363,"threads_started":1}
I20260812 16:44:25.626052 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.601432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:39697 (local address 127.0.0.1:39152)
0812 16:44:25.613838 (+ 12406us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:25.613864 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:25.614038 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:25.617463 (+ 3425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:25.617470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:25.617505 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:25.619087 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.619102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.621066 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.621072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.622958 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.622974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.625144 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.625154 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.625345 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.625382 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:44:25.625499 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 16:44:25.625651 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12212,"mutex_wait_us":25,"thread_start_us":17244,"threads_started":1}
I20260812 16:44:25.626151 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.602103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.28.243.1:39697)
0812 16:44:25.616255 (+ 14152us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.616266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.616305 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:44:25.616434 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.616444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.616782 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.617040 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:25.619341 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.620909 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.623195 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.624674 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.624848 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.625095 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:25.625600 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 16:44:25.625813 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13945,"thread_start_us":13783,"threads_started":1}
I20260812 16:44:25.625463 31247 tablet_service.cc:1467] Processing CreateTablet for tablet b01719a81706491181e57cf63d3c792f (DEFAULT_TABLE table=test-table [id=7abbe2f207d84548859df07e0cde4cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:25.627369 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.602730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:41009 (local address 127.0.0.1:42024)
0812 16:44:25.618268 (+ 15538us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:25.618288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:25.618395 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:25.619876 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:25.619880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:25.619908 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:25.620295 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.620309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.621976 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.621982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.624439 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.624473 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.626548 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.626554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.626741 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.626774 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:44:25.626861 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:44:25.627030 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15424,"mutex_wait_us":1,"thread_start_us":11404,"threads_started":1}
I20260812 16:44:25.627736 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.602793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.28.243.2:41009)
0812 16:44:25.619051 (+ 16258us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.619062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.619101 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:44:25.619213 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.619222 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.619509 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.619882 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:25.620462 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.621795 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.624756 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.626049 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.626211 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:25.626424 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:25.627048 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 16:44:25.627230 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15819,"thread_start_us":16376,"threads_started":1}
I20260812 16:44:25.629606 31247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01719a81706491181e57cf63d3c792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:25.631872 31022 tablet_service.cc:1467] Processing CreateTablet for tablet b01719a81706491181e57cf63d3c792f (DEFAULT_TABLE table=test-table [id=7abbe2f207d84548859df07e0cde4cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:25.632284 30796 tablet_service.cc:1467] Processing CreateTablet for tablet b01719a81706491181e57cf63d3c792f (DEFAULT_TABLE table=test-table [id=7abbe2f207d84548859df07e0cde4cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:25.633590 31022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01719a81706491181e57cf63d3c792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:25.633875 30796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01719a81706491181e57cf63d3c792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:25.647799 31397 tablet_bootstrap.cc:492] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: Bootstrap starting.
I20260812 16:44:25.651310 31399 tablet_bootstrap.cc:492] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: Bootstrap starting.
I20260812 16:44:25.653386 31398 tablet_bootstrap.cc:492] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: Bootstrap starting.
I20260812 16:44:25.653998 31397 tablet_bootstrap.cc:654] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:25.655761 31397 log.cc:824] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:25.657099 31399 tablet_bootstrap.cc:654] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:25.657505 31398 tablet_bootstrap.cc:654] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:25.658895 31399 log.cc:824] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:25.658895 31398 log.cc:824] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:25.659161 31397 tablet_bootstrap.cc:492] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: No bootstrap required, opened a new log
I20260812 16:44:25.659451 31397 ts_tablet_manager.cc:1397] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260812 16:44:25.662649 31399 tablet_bootstrap.cc:492] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: No bootstrap required, opened a new log
I20260812 16:44:25.663102 31399 ts_tablet_manager.cc:1397] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260812 16:44:25.663679 31398 tablet_bootstrap.cc:492] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: No bootstrap required, opened a new log
I20260812 16:44:25.664000 31398 ts_tablet_manager.cc:1397] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260812 16:44:25.669893 31397 raft_consensus.cc:348] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.670418 31397 raft_consensus.cc:374] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:25.670619 31397 raft_consensus.cc:729] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7701001bace4615ac9700ec49276072, State: Initialized, Role: FOLLOWER
I20260812 16:44:25.671492 31397 consensus_queue.cc:260] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.672935 31121 heartbeater.cc:503] Master 127.28.243.62:42257 was elected leader, sending a full tablet report...
I20260812 16:44:25.676680 31398 raft_consensus.cc:348] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.676683 31399 raft_consensus.cc:348] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.677227 31398 raft_consensus.cc:374] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:25.677227 31399 raft_consensus.cc:374] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:25.677342 31398 raft_consensus.cc:729] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e055262d02e24f2c97cb393b30c75bbf, State: Initialized, Role: FOLLOWER
I20260812 16:44:25.677342 31399 raft_consensus.cc:729] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54779cbb495f476893a7e6d408321b70, State: Initialized, Role: FOLLOWER
I20260812 16:44:25.678095 31398 consensus_queue.cc:260] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.678084 31399 consensus_queue.cc:260] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.681599 31397 ts_tablet_manager.cc:1428] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: Time spent starting tablet: real 0.022s user 0.013s sys 0.001s
I20260812 16:44:25.683779 31398 ts_tablet_manager.cc:1428] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20260812 16:44:25.683853 31399 ts_tablet_manager.cc:1428] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260812 16:44:25.684361 31370 heartbeater.cc:503] Master 127.28.243.62:42257 was elected leader, sending a full tablet report...
W20260812 16:44:25.726280 30904 tablet.cc:2369] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:44:25.733925 31372 tablet.cc:2369] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:44:25.847035 31124 tablet.cc:2369] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:44:25.991518 31406 raft_consensus.cc:482] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:25.991838 31406 raft_consensus.cc:504] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:25.994184 31406 leader_election.cc:288] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e055262d02e24f2c97cb393b30c75bbf (127.28.243.3:33077), b7701001bace4615ac9700ec49276072 (127.28.243.2:41009)
I20260812 16:44:26.002336 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.996011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:45149 (local address 127.28.243.2:41009)
0812 16:44:25.996254 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.996259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.996552 (+ 293us) server_negotiation.cc:407] Connection header received
0812 16:44:25.996772 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.996776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.996838 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.996914 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:25.997647 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.998462 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.000466 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.001328 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.001501 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.001585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.001910 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.002058 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 16:44:26.002355 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.994789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:34233 (local address 127.28.243.3:33077)
0812 16:44:25.995038 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:25.995043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:25.995660 (+ 617us) server_negotiation.cc:407] Connection header received
0812 16:44:25.995931 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:25.995938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:25.996038 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:25.996105 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:25.996910 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.997800 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.000060 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.001062 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.001253 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.001340 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.001682 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.001831 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:44:26.002554 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.994644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:33077 (local address 127.28.243.1:34233)
0812 16:44:25.995527 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:25.995547 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:25.995719 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:25.996330 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:25.996333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:25.996356 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:25.996745 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.996755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.997974 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.997980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:25.999859 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.999876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.001314 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.001321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:26.001484 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.001516 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:44:26.001616 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 16:44:26.001775 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":422,"threads_started":1}
I20260812 16:44:26.002707 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:25.995692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:41009 (local address 127.28.243.1:45149)
0812 16:44:25.996432 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:25.996450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:25.996618 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:25.996951 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:25.996956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:25.996983 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:25.997480 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:25.997491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:25.998612 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:25.998616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:26.000279 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:26.000293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.001574 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.001581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:26.001737 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.001774 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:44:26.001864 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 16:44:26.002014 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":1,"thread_start_us":316,"threads_started":1}
I20260812 16:44:26.004436 31050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01719a81706491181e57cf63d3c792f" candidate_uuid: "54779cbb495f476893a7e6d408321b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7701001bace4615ac9700ec49276072" is_pre_election: true
I20260812 16:44:26.004570 31280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01719a81706491181e57cf63d3c792f" candidate_uuid: "54779cbb495f476893a7e6d408321b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e055262d02e24f2c97cb393b30c75bbf" is_pre_election: true
I20260812 16:44:26.005264 31050 raft_consensus.cc:2454] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54779cbb495f476893a7e6d408321b70 in term 0.
I20260812 16:44:26.005558 31280 raft_consensus.cc:2454] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54779cbb495f476893a7e6d408321b70 in term 0.
I20260812 16:44:26.006390 30716 leader_election.cc:302] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54779cbb495f476893a7e6d408321b70, b7701001bace4615ac9700ec49276072; no voters:
I20260812 16:44:26.006865 31406 raft_consensus.cc:2781] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:26.007066 31406 raft_consensus.cc:482] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:26.007194 31406 raft_consensus.cc:3037] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:26.009794 31406 raft_consensus.cc:504] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:26.010454 31406 leader_election.cc:288] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [CANDIDATE]: Term 1 election: Requested vote from peers e055262d02e24f2c97cb393b30c75bbf (127.28.243.3:33077), b7701001bace4615ac9700ec49276072 (127.28.243.2:41009)
I20260812 16:44:26.011512 31050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01719a81706491181e57cf63d3c792f" candidate_uuid: "54779cbb495f476893a7e6d408321b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7701001bace4615ac9700ec49276072"
I20260812 16:44:26.011508 31280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01719a81706491181e57cf63d3c792f" candidate_uuid: "54779cbb495f476893a7e6d408321b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e055262d02e24f2c97cb393b30c75bbf"
I20260812 16:44:26.011818 31050 raft_consensus.cc:3037] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:26.011792 31280 raft_consensus.cc:3037] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:26.016475 31050 raft_consensus.cc:2454] T b01719a81706491181e57cf63d3c792f P b7701001bace4615ac9700ec49276072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54779cbb495f476893a7e6d408321b70 in term 1.
I20260812 16:44:26.016475 31280 raft_consensus.cc:2454] T b01719a81706491181e57cf63d3c792f P e055262d02e24f2c97cb393b30c75bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54779cbb495f476893a7e6d408321b70 in term 1.
I20260812 16:44:26.017190 30716 leader_election.cc:302] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54779cbb495f476893a7e6d408321b70, e055262d02e24f2c97cb393b30c75bbf; no voters:
I20260812 16:44:26.017743 31406 raft_consensus.cc:2781] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:26.018502 31406 raft_consensus.cc:686] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [term 1 LEADER]: Becoming Leader. State: Replica: 54779cbb495f476893a7e6d408321b70, State: Running, Role: LEADER
I20260812 16:44:26.019632 31406 consensus_queue.cc:237] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } }
I20260812 16:44:26.027253 30599 catalog_manager.cc:5495] T b01719a81706491181e57cf63d3c792f P 54779cbb495f476893a7e6d408321b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54779cbb495f476893a7e6d408321b70 (127.28.243.1). New cstate: current_term: 1 leader_uuid: "54779cbb495f476893a7e6d408321b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e055262d02e24f2c97cb393b30c75bbf" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7701001bace4615ac9700ec49276072" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54779cbb495f476893a7e6d408321b70" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 39697 } health_report { overall_health: HEALTHY } } }
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:26.099828 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:26.077854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.28.243.1:39697)
0812 16:44:26.078159 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:26.078166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:26.078479 (+ 313us) server_negotiation.cc:407] Connection header received
0812 16:44:26.078696 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:26.078700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:26.078792 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:26.078883 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:26.080639 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.081546 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.083718 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.084269 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.084456 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.091475 (+ 7019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:26.091504 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:26.091529 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:26.091599 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:26.095168 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.096230 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.096247 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.096250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.096555 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.097249 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.097255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.097258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.097775 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:26.098054 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.099074 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.099431 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:26.176386 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:26.157199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.28.243.1:39697)
0812 16:44:26.157417 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:26.157423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:26.157841 (+ 418us) server_negotiation.cc:407] Connection header received
0812 16:44:26.158043 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:26.158047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:26.158118 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:26.158207 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:26.159943 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.160912 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.162328 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.162813 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.162964 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.168669 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:26.168679 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:26.168684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:26.168752 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:26.172657 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.173770 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.173775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.173777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.173993 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.174469 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.174473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.174476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.174828 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:26.175182 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.175547 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.175865 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:26.245216 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:26.225402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.28.243.1:39697)
0812 16:44:26.225718 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:26.225726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:26.226336 (+ 610us) server_negotiation.cc:407] Connection header received
0812 16:44:26.226843 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:26.226849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:26.227126 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:26.227256 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:26.229898 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.231245 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.233409 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.233913 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.234099 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.238492 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:26.238497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:26.238501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:26.238541 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:26.241240 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.242148 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.242155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.242158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.242334 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.243065 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.243071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.243074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.243405 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:26.243704 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.244548 (+ 844us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.244837 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260812 16:44:26.249184 30762 tablet_service.cc:2908] Rejecting scan request for tablet b01719a81706491181e57cf63d3c792f: Uninitialized: clean time has not yet been initialized
I20260812 16:44:26.254490 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30694
I20260812 16:44:26.272775 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:26.255650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.28.243.1:39697)
0812 16:44:26.255847 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:26.255852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:26.256211 (+ 359us) server_negotiation.cc:407] Connection header received
0812 16:44:26.256427 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:26.256433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:26.256510 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:26.256609 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:26.257982 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.258713 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.260396 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.260963 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.261079 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.263901 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:26.263906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:26.263910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:26.263949 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:26.268198 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.269616 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.269622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.269624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.269843 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.270693 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.270698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.270700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.271237 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:26.271642 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.272159 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.272452 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:44:26.273680 30892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:26.376382 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30694
I20260812 16:44:26.400981 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30912
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:44:26.420379 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:26.401982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.28.243.2:41009)
0812 16:44:26.402184 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:26.402189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:26.402209 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:44:26.402277 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:26.402281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:26.402344 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:26.402429 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:26.404080 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.405542 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.407634 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.408163 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.408303 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.413813 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:26.413832 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:26.413852 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:26.413902 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:26.417181 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.418058 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.418066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.418068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.418234 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.418656 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.418660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.418662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.419145 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:26.419394 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.419804 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.420012 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:44:26.421272 31110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:26.525082 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30912
I20260812 16:44:26.552436 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31135
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:44:26.577491 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:26.553496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.28.243.3:33077)
0812 16:44:26.554066 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:26.554071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:26.554089 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:44:26.554147 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:26.554151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:26.554238 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:26.554334 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:26.555762 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.556514 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.558322 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:26.558848 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:26.559092 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:26.570460 (+ 11368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:26.570481 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:26.570499 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:26.570551 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:26.574336 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.575297 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.575305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.575307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.575469 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:26.575918 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:26.575922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:26.575924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:26.576358 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:26.576599 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:26.576896 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 16:44:26.577107 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":278,"threads_started":1}
I20260812 16:44:26.578550 31359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:26.675648 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31135
I20260812 16:44:26.703734 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30553
I20260812 16:44:26.704856 30646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:26.825371 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30553
2026-08-12T16:44:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5373 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[31446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[31446](info): set up 2 sockets
Aug 12 16:44:26 dist-test-slave-1vmg krb5kdc[31446](info): commencing operation
krb5kdc: starting...
W20260812 16:44:28.134693 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:28 dist-test-slave-1vmg krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:28.384339 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:35717
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:28.768930 31468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:28.769275 31468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:28.769371 31468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:28.777997 31468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:28.778157 31468 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 16:44:28.778203 31468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:28.778264 31468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:28.778308 31468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:28.778373 31468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:28.789742 31468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:35717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:28.791208 31468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:28.792677 31468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:28.809664 31479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:28.809746 31480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:28.809755 31482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:28.810734 31468 server_base.cc:1034] running on GCE node
I20260812 16:44:28.811758 31468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:28.814301 31468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:28.817811 31468 hybrid_clock.cc:648] HybridClock initialized: now 1786553068817641 us; error 97 us; skew 500 ppm
I20260812 16:44:28.818356 31468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:28 dist-test-slave-1vmg krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:28.828473 31468 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:28.831341 31468 webserver.cc:460] Webserver started at http://127.28.243.62:44477/ using document root <none> and password file <none>
I20260812 16:44:28.832516 31468 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:28.832660 31468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:28.833087 31468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:28.837693 31468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4"
format_stamp: "Formatted at 2026-08-12 16:44:28 on dist-test-slave-1vmg"
server_key: "05b30d4a266c0e9c1e3def2f95a05081"
server_key_iv: "d59bba43f4dcfdf3f662ae32a5b4659f"
server_key_version: "encryptionkey@0"
I20260812 16:44:28.838817 31468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4"
format_stamp: "Formatted at 2026-08-12 16:44:28 on dist-test-slave-1vmg"
server_key: "05b30d4a266c0e9c1e3def2f95a05081"
server_key_iv: "d59bba43f4dcfdf3f662ae32a5b4659f"
server_key_version: "encryptionkey@0"
I20260812 16:44:28.846632 31468 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 16:44:28.851996 31492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:28.853737 31468 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:44:28.853937 31468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4"
format_stamp: "Formatted at 2026-08-12 16:44:28 on dist-test-slave-1vmg"
server_key: "05b30d4a266c0e9c1e3def2f95a05081"
server_key_iv: "d59bba43f4dcfdf3f662ae32a5b4659f"
server_key_version: "encryptionkey@0"
I20260812 16:44:28.854218 31468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:28.879261 31468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:28.886303 31468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:28.887295 31468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:28.914655 31468 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:35717
I20260812 16:44:28.914682 31599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:35717 every 8 connection(s)
I20260812 16:44:28.916649 31468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:28.923003 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31468
I20260812 16:44:28.923079 31600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:28.923518 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:28.924515 29644 external_mini_cluster.cc:1428] Setting key 2f9927600c4624b63417c505bf8a7aab
I20260812 16:44:28.940539 31600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: Bootstrap starting.
Aug 12 16:44:28 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:28.945309 31600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:28.946899 31600 log.cc:824] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:28.951094 31600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: No bootstrap required, opened a new log
I20260812 16:44:28.963153 31600 raft_consensus.cc:348] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 35717 } }
I20260812 16:44:28.963970 31600 raft_consensus.cc:374] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:28.964154 31600 raft_consensus.cc:729] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f328b6c7c1e6418494ad34b2dc6ef7e4, State: Initialized, Role: FOLLOWER
I20260812 16:44:28.965052 31600 consensus_queue.cc:260] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 35717 } }
I20260812 16:44:28.965413 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:28.927405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.28.243.62:35717)
0812 16:44:28.928370 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:28.928384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:28.928438 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:44:28.931613 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:28.931671 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:28.932497 (+ 826us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:28.932955 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:28.935038 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:28.936560 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:28.938620 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:28.939607 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:28.939801 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:28.949670 (+ 9869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:28.949685 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:28.949709 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:28.949763 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:28.958413 (+ 8650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:28.959647 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:28.959659 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:28.959670 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:28.959993 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:28.960605 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:28.960611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:28.960614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:28.961540 (+ 926us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:28.961905 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:28.962980 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0812 16:44:28.963621 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":443,"threads_started":1}
I20260812 16:44:28.965421 31600 raft_consensus.cc:388] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:28.965654 31600 raft_consensus.cc:482] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:28.965992 31600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:28.969718 31600 raft_consensus.cc:504] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 35717 } }
I20260812 16:44:28.970487 31600 leader_election.cc:302] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f328b6c7c1e6418494ad34b2dc6ef7e4; no voters:
I20260812 16:44:28.976449 31600 leader_election.cc:288] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:28.976641 31605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:28.977478 31605 raft_consensus.cc:686] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [term 1 LEADER]: Becoming Leader. State: Replica: f328b6c7c1e6418494ad34b2dc6ef7e4, State: Running, Role: LEADER
I20260812 16:44:28.978348 31605 consensus_queue.cc:237] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 35717 } }
I20260812 16:44:28.980931 31600 sys_catalog.cc:564] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:28.981742 31612 sys_catalog.cc:455] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 35717 } } }
I20260812 16:44:28.982717 31612 sys_catalog.cc:458] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:28.987639 31610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f328b6c7c1e6418494ad34b2dc6ef7e4. Latest consensus state: current_term: 1 leader_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f328b6c7c1e6418494ad34b2dc6ef7e4" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 35717 } } }
I20260812 16:44:28.988039 31610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:28.989315 31619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:28.998198 31619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:29.013937 31619 catalog_manager.cc:1347] Generated new cluster ID: bb89ae4ecf0f4332b593b3ff444c90e7
I20260812 16:44:29.014094 31619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:29.079620 31619 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:29.081568 31619 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:29.095013 31619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: Generated new TSK 0
I20260812 16:44:29.095943 31619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:29.164001 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:35717
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:29.544382 31650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:29.544726 31650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:29.544873 31650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:29.553630 31650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:29.553818 31650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:29.553948 31650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:44:29.554066 31650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:29.567427 31650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:29.568957 31650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:29.570433 31650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:29.587491 31663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:29.587632 31661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:29.587607 31665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:29.589001 31650 server_base.cc:1034] running on GCE node
I20260812 16:44:29.589769 31650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:29.590906 31650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:29.592226 31650 hybrid_clock.cc:648] HybridClock initialized: now 1786553069592149 us; error 37 us; skew 500 ppm
I20260812 16:44:29.592600 31650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:29 dist-test-slave-1vmg krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553069, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:29.596339 31650 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:29.597983 31650 webserver.cc:460] Webserver started at http://127.28.243.1:41291/ using document root <none> and password file <none>
I20260812 16:44:29.598776 31650 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:29.598886 31650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:29.599284 31650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:29.602697 31650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "ab4432546da946e8851b4f9cd5499335"
format_stamp: "Formatted at 2026-08-12 16:44:29 on dist-test-slave-1vmg"
server_key: "7910cd287b6a4508444b5d14048f5031"
server_key_iv: "4a394671b2b38505a739408738bbbb30"
server_key_version: "encryptionkey@0"
I20260812 16:44:29.604431 31650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "ab4432546da946e8851b4f9cd5499335"
format_stamp: "Formatted at 2026-08-12 16:44:29 on dist-test-slave-1vmg"
server_key: "7910cd287b6a4508444b5d14048f5031"
server_key_iv: "4a394671b2b38505a739408738bbbb30"
server_key_version: "encryptionkey@0"
I20260812 16:44:29.611462 31650 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:44:29.615461 31675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:29.617398 31650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:44:29.617621 31650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "ab4432546da946e8851b4f9cd5499335"
format_stamp: "Formatted at 2026-08-12 16:44:29 on dist-test-slave-1vmg"
server_key: "7910cd287b6a4508444b5d14048f5031"
server_key_iv: "4a394671b2b38505a739408738bbbb30"
server_key_version: "encryptionkey@0"
I20260812 16:44:29.617887 31650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:29.651320 31650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:29.656649 31650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:29.656931 31650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:29.660154 31650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:29.660252 31650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:29.660394 31650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:29.660450 31650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:44:29 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553069, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:29.688704 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:29.661333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:34513 (local address 127.28.243.62:35717)
0812 16:44:29.661861 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:29.661869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:29.662704 (+ 835us) server_negotiation.cc:407] Connection header received
0812 16:44:29.665645 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:29.665652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:29.665758 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:29.665900 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:29.668838 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.670000 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.673178 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.673933 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.674121 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:29.679822 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:29.679831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:29.679837 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:29.679893 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:29.684000 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:29.685374 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:29.685379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:29.685381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:29.685553 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:29.686242 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:29.686248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:29.686252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:29.686616 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:29.686782 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:29.688006 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0812 16:44:29.688288 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260812 16:44:29.689884 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:29.661465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.28.243.1:34513)
0812 16:44:29.662462 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:29.662528 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:29.665238 (+ 2710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:29.666325 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:29.666342 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:29.667354 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:29.668425 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:29.668459 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.670238 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.670243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:29.672893 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:29.672938 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.674295 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.674306 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:29.674746 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:29.675712 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:29.675738 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:29.679369 (+ 3631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:29.684278 (+ 4909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:29.684293 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:29.684316 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:29.685138 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:29.685704 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:29.685710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:29.685714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:29.686078 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:29.686880 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:29.686938 (+ 58us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:29.687501 (+ 563us) client_negotiation.cc:769] Sending connection context
0812 16:44:29.687891 (+ 390us) client_negotiation.cc:240] Negotiation successful
0812 16:44:29.688520 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":396,"threads_started":1}
I20260812 16:44:29.714126 31650 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:33547
I20260812 16:44:29.714731 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:29.696603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:34365 (local address 127.28.243.62:35717)
0812 16:44:29.696781 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:29.696786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:29.697860 (+ 1074us) server_negotiation.cc:407] Connection header received
0812 16:44:29.698104 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:29.698109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:29.698198 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:29.698301 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:29.700351 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.701127 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.703990 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.704740 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.704946 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:29.707438 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:29.707447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:29.707451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:29.707519 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:29.711377 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:29.712401 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:29.712407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:29.712410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:29.712565 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:29.713067 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:29.713072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:29.713075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:29.713404 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:29.713567 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:29.714089 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 16:44:29.714360 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:44:29.714857 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:29.696711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.28.243.1:34365)
0812 16:44:29.697641 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:29.697671 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:29.697935 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:29.698361 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:29.698365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:29.699319 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:29.700086 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:29.700106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.701327 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.701332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:29.703668 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:29.703696 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:29.705069 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:29.705075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:29.705225 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:29.706042 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:29.706064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:29.707119 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:29.711590 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:29.711596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:29.711599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:29.712221 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:29.712692 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:29.712708 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:29.712710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:29.712896 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:29.713541 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:29.713546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:29.713772 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 16:44:29.713948 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 16:44:29.714399 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":485,"threads_started":1}
I20260812 16:44:29.716351 31650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:29.714135 31892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:33547 every 8 connection(s)
I20260812 16:44:29.719059 31893 heartbeater.cc:347] Connected to a master server at 127.28.243.62:35717
I20260812 16:44:29.719341 31893 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:29.720162 31893 heartbeater.cc:511] Master 127.28.243.62:35717 requested a full tablet report, sending...
I20260812 16:44:29.720713 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31650
I20260812 16:44:29.720950 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:29.721815 29644 external_mini_cluster.cc:1428] Setting key 533ae70251406f226e61773e2ea57a1b
I20260812 16:44:29.723192 31521 ts_manager.cc:194] Registered new tserver with Master: ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547)
I20260812 16:44:29.725216 31521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:34513
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:29.781267 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:35717
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:30.157303 31899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:30.157645 31899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:30.157799 31899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:30.165824 31899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:30.166009 31899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:30.166134 31899 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:44:30.166275 31899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:30.177407 31899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:30.178865 31899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:30.180334 31899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:30.197266 31906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:30.197203 31915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:30.197232 31908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:30.197991 31899 server_base.cc:1034] running on GCE node
I20260812 16:44:30.198750 31899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:30.199869 31899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:30.201244 31899 hybrid_clock.cc:648] HybridClock initialized: now 1786553070201105 us; error 51 us; skew 500 ppm
I20260812 16:44:30.201776 31899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:30 dist-test-slave-1vmg krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:30.206266 31899 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:30.208501 31899 webserver.cc:460] Webserver started at http://127.28.243.2:34465/ using document root <none> and password file <none>
I20260812 16:44:30.209774 31899 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:30.209929 31899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:30.210507 31899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:30.214967 31899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "3b4dd8917579442fb234097fc5c46d54"
format_stamp: "Formatted at 2026-08-12 16:44:30 on dist-test-slave-1vmg"
server_key: "7b8bf7ebe0ed473d3e92719e9ac2cd19"
server_key_iv: "92d48f80687fdce3446835d4e36e252f"
server_key_version: "encryptionkey@0"
I20260812 16:44:30.216012 31899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "3b4dd8917579442fb234097fc5c46d54"
format_stamp: "Formatted at 2026-08-12 16:44:30 on dist-test-slave-1vmg"
server_key: "7b8bf7ebe0ed473d3e92719e9ac2cd19"
server_key_iv: "92d48f80687fdce3446835d4e36e252f"
server_key_version: "encryptionkey@0"
I20260812 16:44:30.223114 31899 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:44:30.228224 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:30.230600 31899 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:44:30.230841 31899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "3b4dd8917579442fb234097fc5c46d54"
format_stamp: "Formatted at 2026-08-12 16:44:30 on dist-test-slave-1vmg"
server_key: "7b8bf7ebe0ed473d3e92719e9ac2cd19"
server_key_iv: "92d48f80687fdce3446835d4e36e252f"
server_key_version: "encryptionkey@0"
I20260812 16:44:30.231171 31899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:30.248963 31899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:30.254535 31899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:30.254807 31899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:30.257905 31899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:30.258033 31899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:30.258200 31899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:30.258281 31899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:44:30 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:30.287577 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.258992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:40989 (local address 127.28.243.62:35717)
0812 16:44:30.259529 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:30.259534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:30.260999 (+ 1465us) server_negotiation.cc:407] Connection header received
0812 16:44:30.263227 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:30.263234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:30.263332 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:30.263442 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:30.265521 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.266618 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.269221 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.269842 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.270001 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.276504 (+ 6503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:30.276512 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:30.276518 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:30.276571 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:30.280897 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.282684 (+ 1787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.282691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.282695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.282890 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.283750 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.283755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.283758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.284185 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:30.284392 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:30.286689 (+ 2297us) server_negotiation.cc:299] Negotiation successful
0812 16:44:30.287104 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":312,"threads_started":1}
I20260812 16:44:30.289403 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.259863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.28.243.2:40989)
0812 16:44:30.260819 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:30.260863 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:30.262839 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:30.263906 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:30.263919 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:30.264562 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:30.265312 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.265336 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.266800 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.266804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.269012 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.269031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.270315 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.270327 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.270725 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.271766 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:30.271810 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:30.276006 (+ 4196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:30.281152 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.281169 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.281201 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.282274 (+ 1073us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.283107 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.283113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.283118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.283561 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.284487 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:30.284505 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:30.285608 (+ 1103us) client_negotiation.cc:769] Sending connection context
0812 16:44:30.286398 (+ 790us) client_negotiation.cc:240] Negotiation successful
0812 16:44:30.287241 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":382,"threads_started":1}
I20260812 16:44:30.310671 31899 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:33391
I20260812 16:44:30.310793 32149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:33391 every 8 connection(s)
I20260812 16:44:30.312355 31899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:30.315192 32150 heartbeater.cc:347] Connected to a master server at 127.28.243.62:35717
I20260812 16:44:30.316354 32150 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:30.317143 32150 heartbeater.cc:511] Master 127.28.243.62:35717 requested a full tablet report, sending...
I20260812 16:44:30.318782 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.299322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:59039 (local address 127.28.243.62:35717)
0812 16:44:30.299561 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:30.299568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:30.300112 (+ 544us) server_negotiation.cc:407] Connection header received
0812 16:44:30.300338 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:30.300344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:30.300434 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:30.300545 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:30.302590 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.304572 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.307397 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.308383 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.308627 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.311860 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:30.311869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:30.311873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:30.311931 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:30.315726 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.316618 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.316631 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.316634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.316773 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.317159 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.317163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.317165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.317547 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:30.317717 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:30.318207 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 16:44:30.318452 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 16:44:30.318809 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.298609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.28.243.2:59039)
0812 16:44:30.299994 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:30.300016 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:30.300202 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:30.300648 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:30.300655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:30.301493 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:30.302140 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.302163 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.304839 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.304845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.307055 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.307083 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.308770 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.308781 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.308998 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.309926 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:30.309958 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:30.311395 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:30.315898 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.315903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.315906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.316479 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.316899 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.316904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.316906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.317040 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.317674 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:30.317679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:30.317873 (+ 194us) client_negotiation.cc:769] Sending connection context
0812 16:44:30.318028 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 16:44:30.318334 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"thread_start_us":1031,"threads_started":1}
I20260812 16:44:30.318830 31521 ts_manager.cc:194] Registered new tserver with Master: 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391)
I20260812 16:44:30.319028 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31899
I20260812 16:44:30.319232 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:30.319789 29644 external_mini_cluster.cc:1428] Setting key 51a1ddc1cac76d1714b85bb4b0e8e733
I20260812 16:44:30.319968 31521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:40989
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:30.374378 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:35717
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:30.728555 31893 heartbeater.cc:503] Master 127.28.243.62:35717 was elected leader, sending a full tablet report...
W20260812 16:44:30.737914 32164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:30.738243 32164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:30.738373 32164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:30.746151 32164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:30.746321 32164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:30.746418 32164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:44:30.746546 32164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:30.757380 32164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:35717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:30.758873 32164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:30.760460 32164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:30.775568 32175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:30.775599 32173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:30.775568 32172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:30.776592 32164 server_base.cc:1034] running on GCE node
I20260812 16:44:30.777330 32164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:30.778473 32164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:30.779781 32164 hybrid_clock.cc:648] HybridClock initialized: now 1786553070779700 us; error 39 us; skew 500 ppm
I20260812 16:44:30.780143 32164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:30 dist-test-slave-1vmg krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:30.784782 32164 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:30.786612 32164 webserver.cc:460] Webserver started at http://127.28.243.3:37629/ using document root <none> and password file <none>
I20260812 16:44:30.787544 32164 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:30.787669 32164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:30.788035 32164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:30.791611 32164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "4b998c59676b4507855ba921d0ad817b"
format_stamp: "Formatted at 2026-08-12 16:44:30 on dist-test-slave-1vmg"
server_key: "bd6c05c502a53d28564406efd08556de"
server_key_iv: "23c1d3a4b7941670772962503febe531"
server_key_version: "encryptionkey@0"
I20260812 16:44:30.792536 32164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "4b998c59676b4507855ba921d0ad817b"
format_stamp: "Formatted at 2026-08-12 16:44:30 on dist-test-slave-1vmg"
server_key: "bd6c05c502a53d28564406efd08556de"
server_key_iv: "23c1d3a4b7941670772962503febe531"
server_key_version: "encryptionkey@0"
I20260812 16:44:30.798624 32164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:44:30.802754 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:30.804715 32164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:44:30.804937 32164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "4b998c59676b4507855ba921d0ad817b"
format_stamp: "Formatted at 2026-08-12 16:44:30 on dist-test-slave-1vmg"
server_key: "bd6c05c502a53d28564406efd08556de"
server_key_iv: "23c1d3a4b7941670772962503febe531"
server_key_version: "encryptionkey@0"
I20260812 16:44:30.805222 32164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:30.833482 32164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:30.838037 32164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:30.838330 32164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:30.841254 32164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:30.841353 32164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:30.841495 32164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:30.841563 32164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:44:30 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:30.870549 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.841823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:38213 (local address 127.28.243.62:35717)
0812 16:44:30.842459 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:30.842466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:30.843608 (+ 1142us) server_negotiation.cc:407] Connection header received
0812 16:44:30.847215 (+ 3607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:30.847222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:30.847318 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:30.847436 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:30.849859 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.851149 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.853804 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.854501 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.854644 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.860851 (+ 6207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:30.860859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:30.860865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:30.860915 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:30.864946 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.866284 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.866291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.866294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.866458 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.867168 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.867174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.867178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.867615 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:30.867771 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:30.869753 (+ 1982us) server_negotiation.cc:299] Negotiation successful
0812 16:44:30.870075 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":373,"threads_started":1}
I20260812 16:44:30.872474 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.842459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.28.243.3:38213)
0812 16:44:30.843413 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:30.843465 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:30.846501 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:30.847735 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:30.847751 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:30.848595 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:30.849625 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.849654 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.851434 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.851444 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.853523 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.853550 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.854809 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.854817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.855221 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.856208 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:30.856234 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:30.860470 (+ 4236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:30.865249 (+ 4779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.865263 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.865292 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.866119 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.866583 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.866589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.866593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.867031 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.867763 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:30.867773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:30.868918 (+ 1145us) client_negotiation.cc:769] Sending connection context
0812 16:44:30.869602 (+ 684us) client_negotiation.cc:240] Negotiation successful
0812 16:44:30.870301 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":400,"threads_started":1}
I20260812 16:44:30.897428 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.879827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:33075 (local address 127.28.243.62:35717)
0812 16:44:30.880082 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:30.880090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:30.881286 (+ 1196us) server_negotiation.cc:407] Connection header received
0812 16:44:30.881582 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:30.881588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:30.881685 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:30.881806 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:30.883511 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.884632 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.887286 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.887987 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.888136 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.890455 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:30.890463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:30.890469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:30.890508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:30.894183 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.895240 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.895246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.895250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.895433 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.895903 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.895909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.895912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.896221 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:30.896379 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:30.896821 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 16:44:30.897062 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:44:30.897567 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.880062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.28.243.3:33075)
0812 16:44:30.881095 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:30.881127 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:30.881351 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:30.882020 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:30.882027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:30.882801 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:30.883314 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.883328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.885021 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.885028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.886997 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:30.887023 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.888236 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.888243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:30.888412 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.889098 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:30.889118 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:30.889975 (+ 857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:30.894379 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.894387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.894392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.895058 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.895575 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:30.895580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:30.895584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:30.895778 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:30.896350 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:30.896355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:30.896560 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 16:44:30.896687 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 16:44:30.897087 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":368,"threads_started":1}
I20260812 16:44:30.901525 32164 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:39111
I20260812 16:44:30.901564 32404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:39111 every 8 connection(s)
I20260812 16:44:30.902861 32164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:30.905771 32405 heartbeater.cc:347] Connected to a master server at 127.28.243.62:35717
I20260812 16:44:30.906268 32405 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:30.907464 32405 heartbeater.cc:511] Master 127.28.243.62:35717 requested a full tablet report, sending...
I20260812 16:44:30.909770 31521 ts_manager.cc:194] Registered new tserver with Master: 4b998c59676b4507855ba921d0ad817b (127.28.243.3:39111)
I20260812 16:44:30.910766 31521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:38213
I20260812 16:44:30.913228 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 32164
I20260812 16:44:30.913455 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:30.914067 29644 external_mini_cluster.cc:1428] Setting key 97462fef288f17027c6e2cc5faaf7cf4
I20260812 16:44:30.917809 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:44:30 dist-test-slave-1vmg krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:44:30 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:30.977407 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:30.959049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.28.243.62:35717)
0812 16:44:30.959282 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:30.959288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:30.959462 (+ 174us) server_negotiation.cc:407] Connection header received
0812 16:44:30.959701 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:30.959706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:30.959788 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:30.959882 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:30.961399 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.962147 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.963701 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:30.964138 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:30.964262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:30.969873 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:30.969882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:30.969888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:30.969975 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:30.973486 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.974518 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.974523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.974525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.974730 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:30.975341 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:30.975346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:30.975348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:30.975714 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:30.975984 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:30.976891 (+ 907us) server_negotiation.cc:299] Negotiation successful
0812 16:44:30.977125 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:44:30.983342 31521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:44:30.994320 31521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:31.033991 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.012194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.28.243.1:33547)
0812 16:44:31.023682 (+ 11488us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.023693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.023731 (+ 38us) server_negotiation.cc:407] Connection header received
0812 16:44:31.023860 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.023869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.024172 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.025506 (+ 1334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.025822 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.027516 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.031331 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.032638 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.032792 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.033025 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.033513 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.033691 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11278,"thread_start_us":12166,"threads_started":1}
I20260812 16:44:31.034047 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.011288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:33547 (local address 127.0.0.1:33912)
0812 16:44:31.012477 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.012540 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.012848 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.024713 (+ 11865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.024723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.024762 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.025264 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.025277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.029191 (+ 3914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.029197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.031123 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.031138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.032903 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.032911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.033192 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.033244 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.033387 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.033617 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":1,"thread_start_us":2294,"threads_started":1}
I20260812 16:44:31.036955 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.013614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:39111 (local address 127.0.0.1:40732)
0812 16:44:31.027601 (+ 13987us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.027625 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.027770 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.028817 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.028823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.028852 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.029450 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.029463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.031836 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.031843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.033785 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.033799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.036144 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.036150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.036364 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.036397 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.036490 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.036644 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13819,"thread_start_us":14249,"threads_started":1}
I20260812 16:44:31.039443 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.014275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.28.243.3:39111)
0812 16:44:31.028210 (+ 13935us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.028217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.028245 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:44:31.028340 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.028346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.028569 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.028818 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.029755 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.031670 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.033994 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.035644 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.035767 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.035997 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.038877 (+ 2880us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.039108 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13755,"thread_start_us":13575,"threads_started":1}
I20260812 16:44:31.038877 31778 tablet_service.cc:1467] Processing CreateTablet for tablet 86eb8064a8f94f7cbd2d766aaab0e228 (DEFAULT_TABLE table=test-table [id=4e48c669aa5d4e94acbb2303a91ed5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:31.040547 31778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86eb8064a8f94f7cbd2d766aaab0e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.041054 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.011436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:33391 (local address 127.0.0.1:45868)
0812 16:44:31.020748 (+ 9312us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.020774 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.020936 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.034153 (+ 13217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.034158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.034191 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.034555 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.034568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.036539 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.036545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.038609 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.038627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.040225 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.040233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.040414 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.040449 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.040570 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.040720 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9097,"thread_start_us":552,"threads_started":1}
I20260812 16:44:31.041661 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.011726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.28.243.2:33391)
0812 16:44:31.033472 (+ 21746us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.033481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.033511 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:44:31.033629 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.033638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.033910 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.034805 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.035031 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.036392 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.038801 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.039970 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.040135 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.040336 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.040632 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.041368 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21570,"thread_start_us":21423,"threads_started":1}
I20260812 16:44:31.044746 32285 tablet_service.cc:1467] Processing CreateTablet for tablet 86eb8064a8f94f7cbd2d766aaab0e228 (DEFAULT_TABLE table=test-table [id=4e48c669aa5d4e94acbb2303a91ed5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:31.047784 32285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86eb8064a8f94f7cbd2d766aaab0e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.047844 32030 tablet_service.cc:1467] Processing CreateTablet for tablet 86eb8064a8f94f7cbd2d766aaab0e228 (DEFAULT_TABLE table=test-table [id=4e48c669aa5d4e94acbb2303a91ed5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:31.049391 32030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86eb8064a8f94f7cbd2d766aaab0e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.053848 32438 tablet_bootstrap.cc:492] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Bootstrap starting.
I20260812 16:44:31.057868 32438 tablet_bootstrap.cc:654] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.059303 32438 log.cc:824] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:31.062423 32438 tablet_bootstrap.cc:492] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: No bootstrap required, opened a new log
I20260812 16:44:31.062798 32438 ts_tablet_manager.cc:1397] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 16:44:31.066468 32444 tablet_bootstrap.cc:492] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Bootstrap starting.
I20260812 16:44:31.069394 32446 tablet_bootstrap.cc:492] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Bootstrap starting.
I20260812 16:44:31.071568 32444 tablet_bootstrap.cc:654] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.073465 32444 log.cc:824] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:31.073812 32438 raft_consensus.cc:348] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.074630 32438 raft_consensus.cc:374] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.074776 32438 raft_consensus.cc:729] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4432546da946e8851b4f9cd5499335, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.074899 32446 tablet_bootstrap.cc:654] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.075565 32438 consensus_queue.cc:260] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.076715 32446 log.cc:824] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:31.077643 32438 ts_tablet_manager.cc:1428] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260812 16:44:31.078744 32444 tablet_bootstrap.cc:492] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: No bootstrap required, opened a new log
I20260812 16:44:31.079167 32444 ts_tablet_manager.cc:1397] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260812 16:44:31.082531 32446 tablet_bootstrap.cc:492] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: No bootstrap required, opened a new log
I20260812 16:44:31.082896 32446 ts_tablet_manager.cc:1397] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260812 16:44:31.088737 32444 raft_consensus.cc:348] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.089308 32444 raft_consensus.cc:374] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.089411 32444 raft_consensus.cc:729] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b998c59676b4507855ba921d0ad817b, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.090179 32444 consensus_queue.cc:260] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.093762 32405 heartbeater.cc:503] Master 127.28.243.62:35717 was elected leader, sending a full tablet report...
I20260812 16:44:31.094307 32444 ts_tablet_manager.cc:1428] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260812 16:44:31.095008 32446 raft_consensus.cc:348] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.095541 32446 raft_consensus.cc:374] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.095667 32446 raft_consensus.cc:729] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4dd8917579442fb234097fc5c46d54, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.096499 32446 consensus_queue.cc:260] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.097810 32150 heartbeater.cc:503] Master 127.28.243.62:35717 was elected leader, sending a full tablet report...
I20260812 16:44:31.098487 32446 ts_tablet_manager.cc:1428] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260812 16:44:31.155002 32406 tablet.cc:2369] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:44:31.174134 32455 raft_consensus.cc:482] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:31.174472 32455 raft_consensus.cc:504] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.176422 32455 leader_election.cc:288] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547), 4b998c59676b4507855ba921d0ad817b (127.28.243.3:39111)
I20260812 16:44:31.183804 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.177406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:38691 (local address 127.28.243.1:33547)
0812 16:44:31.177675 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.177682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.177978 (+ 296us) server_negotiation.cc:407] Connection header received
0812 16:44:31.178195 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.178199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.178264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.178343 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.179235 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.179970 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.182008 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.182842 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.183058 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.183142 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.183419 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.183555 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:44:31.184409 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.176989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:33547 (local address 127.28.243.2:38691)
0812 16:44:31.177851 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.177868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.178049 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.178388 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.178393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.178420 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.179071 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.179085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.180111 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.180115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.181829 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.181846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.183090 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.183097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.183259 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.183286 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.183375 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.183529 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":38,"thread_start_us":514,"threads_started":1}
I20260812 16:44:31.184685 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.177174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:39111 (local address 127.28.243.2:35867)
0812 16:44:31.178259 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.178278 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.178412 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.178745 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.178749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.178773 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.179364 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.179375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.180537 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.180543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.182236 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.182249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.183518 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.183524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.183674 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.183698 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.183779 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.183927 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":23,"thread_start_us":756,"threads_started":1}
I20260812 16:44:31.185289 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.176909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:35867 (local address 127.28.243.3:39111)
0812 16:44:31.177148 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.177155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.178374 (+ 1219us) server_negotiation.cc:407] Connection header received
0812 16:44:31.178557 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.178561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.178631 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.178703 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.179524 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.180391 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.182398 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.183295 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.183454 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.183529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.183822 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.183959 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 16:44:31.186336 31814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb8064a8f94f7cbd2d766aaab0e228" candidate_uuid: "3b4dd8917579442fb234097fc5c46d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4432546da946e8851b4f9cd5499335" is_pre_election: true
I20260812 16:44:31.186620 32323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb8064a8f94f7cbd2d766aaab0e228" candidate_uuid: "3b4dd8917579442fb234097fc5c46d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b998c59676b4507855ba921d0ad817b" is_pre_election: true
I20260812 16:44:31.187311 31814 raft_consensus.cc:2454] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b4dd8917579442fb234097fc5c46d54 in term 0.
I20260812 16:44:31.187381 32323 raft_consensus.cc:2454] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b4dd8917579442fb234097fc5c46d54 in term 0.
I20260812 16:44:31.188431 31927 leader_election.cc:302] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4dd8917579442fb234097fc5c46d54, ab4432546da946e8851b4f9cd5499335; no voters:
I20260812 16:44:31.188949 32455 raft_consensus.cc:2781] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:31.189196 32455 raft_consensus.cc:482] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:31.189361 32455 raft_consensus.cc:3037] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:31.192625 32455 raft_consensus.cc:504] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.193804 32455 leader_election.cc:288] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [CANDIDATE]: Term 1 election: Requested vote from peers ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547), 4b998c59676b4507855ba921d0ad817b (127.28.243.3:39111)
I20260812 16:44:31.194432 31814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb8064a8f94f7cbd2d766aaab0e228" candidate_uuid: "3b4dd8917579442fb234097fc5c46d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4432546da946e8851b4f9cd5499335"
I20260812 16:44:31.194450 32323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb8064a8f94f7cbd2d766aaab0e228" candidate_uuid: "3b4dd8917579442fb234097fc5c46d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b998c59676b4507855ba921d0ad817b"
I20260812 16:44:31.194718 32323 raft_consensus.cc:3037] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:31.194710 31814 raft_consensus.cc:3037] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:31.197268 31814 raft_consensus.cc:2454] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4dd8917579442fb234097fc5c46d54 in term 1.
I20260812 16:44:31.197300 32323 raft_consensus.cc:2454] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b4dd8917579442fb234097fc5c46d54 in term 1.
I20260812 16:44:31.197834 31932 leader_election.cc:302] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4dd8917579442fb234097fc5c46d54, 4b998c59676b4507855ba921d0ad817b; no voters:
I20260812 16:44:31.198225 32455 raft_consensus.cc:2781] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:31.198770 32455 raft_consensus.cc:686] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4dd8917579442fb234097fc5c46d54, State: Running, Role: LEADER
I20260812 16:44:31.199440 32455 consensus_queue.cc:237] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.205093 31520 catalog_manager.cc:5495] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2). New cstate: current_term: 1 leader_uuid: "3b4dd8917579442fb234097fc5c46d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } health_report { overall_health: UNKNOWN } } }
W20260812 16:44:31.219538 31894 tablet.cc:2369] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:44:31.243898 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.237059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.28.243.2:33391)
0812 16:44:31.237335 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.237341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.237524 (+ 183us) server_negotiation.cc:407] Connection header received
0812 16:44:31.237958 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.237965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.238077 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.238211 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:31.238638 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.239549 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.241088 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.241527 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.241648 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.242261 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:31.243019 (+ 758us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:31.243160 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.243544 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.243632 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 16:44:31.258409 32323 raft_consensus.cc:1261] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Refusing update from remote peer 3b4dd8917579442fb234097fc5c46d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:31.260033 32459 consensus_queue.cc:1035] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:31.260833 31814 raft_consensus.cc:1261] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Refusing update from remote peer 3b4dd8917579442fb234097fc5c46d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:31.261958 32459 consensus_queue.cc:1035] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:31.307768 31521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:31.308373 31521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:31.313508 31521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: Sending DeleteTablet for 3 replicas of tablet 86eb8064a8f94f7cbd2d766aaab0e228
I20260812 16:44:31.315310 32030 tablet_service.cc:1514] Processing DeleteTablet for tablet 86eb8064a8f94f7cbd2d766aaab0e228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:31 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:45868
I20260812 16:44:31.315321 31778 tablet_service.cc:1514] Processing DeleteTablet for tablet 86eb8064a8f94f7cbd2d766aaab0e228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:31 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:33912
I20260812 16:44:31.318696 32486 tablet_replica.cc:330] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: stopping tablet replica
I20260812 16:44:31.319416 32486 raft_consensus.cc:2229] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:31.320180 32285 tablet_service.cc:1514] Processing DeleteTablet for tablet 86eb8064a8f94f7cbd2d766aaab0e228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:31 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:40732
I20260812 16:44:31.322234 31521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:44:31.323462 31521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:31.324986 32486 raft_consensus.cc:2258] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:31.333760 32486 ts_tablet_manager.cc:1905] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:31.338120 31778 tablet_service.cc:1467] Processing CreateTablet for tablet 72f059fa5873435885d15216b363324e (DEFAULT_TABLE table=test-table [id=bda432ef472047ea9faf2e18dcb31b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:31.338186 32284 tablet_service.cc:1467] Processing CreateTablet for tablet 72f059fa5873435885d15216b363324e (DEFAULT_TABLE table=test-table [id=bda432ef472047ea9faf2e18dcb31b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:31.338681 31778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f059fa5873435885d15216b363324e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.338773 32284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f059fa5873435885d15216b363324e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.340323 32484 tablet_replica.cc:330] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: stopping tablet replica
I20260812 16:44:31.341001 32484 raft_consensus.cc:2229] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:44:31.342299 32484 raft_consensus.cc:2258] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:31.345350 32030 tablet_service.cc:1467] Processing CreateTablet for tablet 72f059fa5873435885d15216b363324e (DEFAULT_TABLE table=test-table [id=bda432ef472047ea9faf2e18dcb31b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:31.346467 32030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f059fa5873435885d15216b363324e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.346686 32444 tablet_bootstrap.cc:492] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Bootstrap starting.
I20260812 16:44:31.347353 32438 tablet_bootstrap.cc:492] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Bootstrap starting.
I20260812 16:44:31.348451 32444 tablet_bootstrap.cc:654] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.350169 32438 tablet_bootstrap.cc:654] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.354508 32484 ts_tablet_manager.cc:1905] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:31.356561 32446 tablet_bootstrap.cc:492] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Bootstrap starting.
I20260812 16:44:31.358736 32446 tablet_bootstrap.cc:654] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.363404 32444 tablet_bootstrap.cc:492] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: No bootstrap required, opened a new log
I20260812 16:44:31.363718 32444 ts_tablet_manager.cc:1397] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20260812 16:44:31.364840 32444 raft_consensus.cc:348] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.365203 32444 raft_consensus.cc:374] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.365360 32444 raft_consensus.cc:729] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b998c59676b4507855ba921d0ad817b, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.365799 32444 consensus_queue.cc:260] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.367856 32444 ts_tablet_manager.cc:1428] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 16:44:31.368840 32446 tablet_bootstrap.cc:492] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: No bootstrap required, opened a new log
I20260812 16:44:31.369119 32446 ts_tablet_manager.cc:1397] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260812 16:44:31.370255 32484 ts_tablet_manager.cc:1918] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:44:31.370455 32446 raft_consensus.cc:348] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.370728 32446 raft_consensus.cc:374] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.371021 32446 raft_consensus.cc:729] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4dd8917579442fb234097fc5c46d54, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.371485 32446 consensus_queue.cc:260] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.371034 32484 log.cc:1196] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/86eb8064a8f94f7cbd2d766aaab0e228
I20260812 16:44:31.372200 32438 tablet_bootstrap.cc:492] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: No bootstrap required, opened a new log
I20260812 16:44:31.372462 32438 ts_tablet_manager.cc:1397] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20260812 16:44:31.372953 32484 ts_tablet_manager.cc:1939] T 86eb8064a8f94f7cbd2d766aaab0e228 P 3b4dd8917579442fb234097fc5c46d54: Deleting consensus metadata
I20260812 16:44:31.373749 32438 raft_consensus.cc:348] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.374075 32438 raft_consensus.cc:374] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.374189 32438 raft_consensus.cc:729] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4432546da946e8851b4f9cd5499335, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.374351 32446 ts_tablet_manager.cc:1428] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260812 16:44:31.374646 32438 consensus_queue.cc:260] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.375917 31497 catalog_manager.cc:4841] TS 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391): tablet 86eb8064a8f94f7cbd2d766aaab0e228 (table test-table [id=4e48c669aa5d4e94acbb2303a91ed5bb]) successfully deleted
I20260812 16:44:31.376274 32438 ts_tablet_manager.cc:1428] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 16:44:31.377544 32487 tablet_replica.cc:330] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: stopping tablet replica
I20260812 16:44:31.378226 32487 raft_consensus.cc:2229] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:31.378895 32487 raft_consensus.cc:2258] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:31.381745 32487 ts_tablet_manager.cc:1905] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:31.384500 32486 ts_tablet_manager.cc:1918] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:44:31.384775 32486 log.cc:1196] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/86eb8064a8f94f7cbd2d766aaab0e228
I20260812 16:44:31.385461 32486 ts_tablet_manager.cc:1939] T 86eb8064a8f94f7cbd2d766aaab0e228 P ab4432546da946e8851b4f9cd5499335: Deleting consensus metadata
I20260812 16:44:31.388964 31496 catalog_manager.cc:4841] TS ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547): tablet 86eb8064a8f94f7cbd2d766aaab0e228 (table test-table [id=4e48c669aa5d4e94acbb2303a91ed5bb]) successfully deleted
I20260812 16:44:31.390743 32487 ts_tablet_manager.cc:1918] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:44:31.391484 32487 log.cc:1196] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/86eb8064a8f94f7cbd2d766aaab0e228
I20260812 16:44:31.392500 32487 ts_tablet_manager.cc:1939] T 86eb8064a8f94f7cbd2d766aaab0e228 P 4b998c59676b4507855ba921d0ad817b: Deleting consensus metadata
I20260812 16:44:31.394687 31498 catalog_manager.cc:4841] TS 4b998c59676b4507855ba921d0ad817b (127.28.243.3:39111): tablet 86eb8064a8f94f7cbd2d766aaab0e228 (table test-table [id=4e48c669aa5d4e94acbb2303a91ed5bb]) successfully deleted
W20260812 16:44:31.567124 32153 tablet.cc:2369] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:44:31.647271 32452 raft_consensus.cc:482] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:31.647761 32452 raft_consensus.cc:504] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.649966 32452 leader_election.cc:288] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547), 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391)
I20260812 16:44:31.660408 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.650639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:50867 (local address 127.28.243.1:33547)
0812 16:44:31.650836 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.650842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.652138 (+ 1296us) server_negotiation.cc:407] Connection header received
0812 16:44:31.652548 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.652555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.652676 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.652816 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.654182 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.655307 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.657548 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.658969 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.659181 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.659266 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.659930 (+ 664us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.660094 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:44:31.660476 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.650539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:58579 (local address 127.28.243.2:33391)
0812 16:44:31.650804 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.650811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.651819 (+ 1008us) server_negotiation.cc:407] Connection header received
0812 16:44:31.652208 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.652215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.652425 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.652524 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:31.653100 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.654315 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.657678 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.659161 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.659328 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.659400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.659980 (+ 580us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.660130 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 16:44:31.660504 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.650244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:33391 (local address 127.28.243.3:58579)
0812 16:44:31.651606 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.651640 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.651923 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.652569 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.652574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.652603 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.652906 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.652918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.654564 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.654570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.657362 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.657384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.659582 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.659589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.659802 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.659837 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.659931 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.660091 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"mutex_wait_us":21,"thread_start_us":884,"threads_started":1}
I20260812 16:44:31.661635 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.650244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:33547 (local address 127.28.243.3:50867)
0812 16:44:31.651880 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.651908 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.652281 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.653074 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.653081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.653122 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:31.653868 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.653890 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.655476 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.655480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.657332 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.657352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.659236 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.659245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.659567 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.659627 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.661075 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.661243 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"mutex_wait_us":643,"thread_start_us":623,"threads_started":1}
I20260812 16:44:31.661404 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f059fa5873435885d15216b363324e" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4dd8917579442fb234097fc5c46d54" is_pre_election: true
I20260812 16:44:31.662019 32059 raft_consensus.cc:2454] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 0.
I20260812 16:44:31.663071 31814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f059fa5873435885d15216b363324e" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4432546da946e8851b4f9cd5499335" is_pre_election: true
I20260812 16:44:31.663414 31814 raft_consensus.cc:2454] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 0.
I20260812 16:44:31.663216 32198 leader_election.cc:302] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4dd8917579442fb234097fc5c46d54, 4b998c59676b4507855ba921d0ad817b; no voters:
I20260812 16:44:31.664494 32452 raft_consensus.cc:2781] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:31.664778 32452 raft_consensus.cc:482] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:31.664921 32452 raft_consensus.cc:3037] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:31.668520 32452 raft_consensus.cc:504] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.669929 32452 leader_election.cc:288] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 election: Requested vote from peers ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547), 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391)
I20260812 16:44:31.670704 31814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f059fa5873435885d15216b363324e" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4432546da946e8851b4f9cd5499335"
I20260812 16:44:31.670744 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f059fa5873435885d15216b363324e" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4dd8917579442fb234097fc5c46d54"
I20260812 16:44:31.671011 31814 raft_consensus.cc:3037] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:31.671082 32059 raft_consensus.cc:3037] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:31.673579 31814 raft_consensus.cc:2454] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 1.
I20260812 16:44:31.674530 32197 leader_election.cc:302] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b998c59676b4507855ba921d0ad817b, ab4432546da946e8851b4f9cd5499335; no voters:
I20260812 16:44:31.675002 32452 raft_consensus.cc:2781] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:31.675408 32059 raft_consensus.cc:2454] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 1.
I20260812 16:44:31.675788 32452 raft_consensus.cc:686] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 1 LEADER]: Becoming Leader. State: Replica: 4b998c59676b4507855ba921d0ad817b, State: Running, Role: LEADER
I20260812 16:44:31.676267 32452 consensus_queue.cc:237] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.680814 31516 catalog_manager.cc:5495] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b998c59676b4507855ba921d0ad817b (127.28.243.3). New cstate: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } health_report { overall_health: HEALTHY } } }
Aug 12 16:44:31 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:31.780674 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.754351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:35717 (local address 127.0.0.1:33592)
0812 16:44:31.755398 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:31.755429 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:31.755654 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:31.756071 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:31.756076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:31.757190 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:31.757544 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.757563 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.759011 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.759017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.760832 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:31.760855 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.761789 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.761797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:31.761998 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.768099 (+ 6101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:31.768146 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:31.772966 (+ 4820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:31.777316 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:31.777342 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:31.777354 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:31.778267 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:31.778729 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:31.778735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:31.778739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:31.778879 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:31.779482 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:31.779491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:31.779659 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 16:44:31.779836 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 16:44:31.780163 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":471,"threads_started":1}
I20260812 16:44:31.780714 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:31.754486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.28.243.62:35717)
0812 16:44:31.755709 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:31.755714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:31.755738 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:44:31.755790 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:31.755794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:31.755855 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:31.755977 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:31.757861 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.758788 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.761075 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:31.761565 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:31.761681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:31.773337 (+ 11656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:31.773344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:31.773348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:31.773391 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:31.777036 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:31.778446 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:31.778451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:31.778453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:31.778602 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:31.779019 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:31.779025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:31.779028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:31.779345 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:31.779489 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:31.779961 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 16:44:31.780329 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":1057,"thread_start_us":896,"threads_started":1}
I20260812 16:44:31.790251 31516 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:33592:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 16:44:31.792778 31516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:31.802047 31778 tablet_service.cc:1467] Processing CreateTablet for tablet 69c61c39ddce4300b9a18dbebbeca1c9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f295eb32e4ab45c59791f94e83e8e4a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:44:31.802759 31778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c61c39ddce4300b9a18dbebbeca1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.802611 32030 tablet_service.cc:1467] Processing CreateTablet for tablet 69c61c39ddce4300b9a18dbebbeca1c9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f295eb32e4ab45c59791f94e83e8e4a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:44:31.802713 32285 tablet_service.cc:1467] Processing CreateTablet for tablet 69c61c39ddce4300b9a18dbebbeca1c9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f295eb32e4ab45c59791f94e83e8e4a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:44:31.803292 32030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c61c39ddce4300b9a18dbebbeca1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.803370 32285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c61c39ddce4300b9a18dbebbeca1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:31.814162 32438 tablet_bootstrap.cc:492] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: Bootstrap starting.
I20260812 16:44:31.814169 32444 tablet_bootstrap.cc:492] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: Bootstrap starting.
I20260812 16:44:31.816047 32438 tablet_bootstrap.cc:654] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.816282 32444 tablet_bootstrap.cc:654] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.818583 32438 tablet_bootstrap.cc:492] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: No bootstrap required, opened a new log
I20260812 16:44:31.818797 32438 ts_tablet_manager.cc:1397] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 16:44:31.819593 32438 raft_consensus.cc:348] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.819911 32438 raft_consensus.cc:374] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.820003 32438 raft_consensus.cc:729] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4432546da946e8851b4f9cd5499335, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.820290 32438 consensus_queue.cc:260] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.820855 32450 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:31.821256 32450 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:31.822146 32438 ts_tablet_manager.cc:1428] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 16:44:31.826225 32444 tablet_bootstrap.cc:492] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: No bootstrap required, opened a new log
I20260812 16:44:31.826936 32446 tablet_bootstrap.cc:492] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: Bootstrap starting.
I20260812 16:44:31.827631 32444 ts_tablet_manager.cc:1397] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260812 16:44:31.829000 32444 raft_consensus.cc:348] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.829257 32444 raft_consensus.cc:374] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.829339 32444 raft_consensus.cc:729] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b998c59676b4507855ba921d0ad817b, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.829512 32446 tablet_bootstrap.cc:654] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:31.829645 32444 consensus_queue.cc:260] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.830564 32444 ts_tablet_manager.cc:1428] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 16:44:31.830358 32452 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:31.830753 32452 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:31.833889 32446 tablet_bootstrap.cc:492] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: No bootstrap required, opened a new log
I20260812 16:44:31.834151 32446 ts_tablet_manager.cc:1397] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 16:44:31.835253 32446 raft_consensus.cc:348] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.835647 32446 raft_consensus.cc:374] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:31.835798 32446 raft_consensus.cc:729] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4dd8917579442fb234097fc5c46d54, State: Initialized, Role: FOLLOWER
I20260812 16:44:31.836130 32446 consensus_queue.cc:260] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:31.836745 32474 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:31.836997 32446 ts_tablet_manager.cc:1428] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 16:44:31.837144 32474 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:32.035195 32452 raft_consensus.cc:482] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:32.035470 32452 raft_consensus.cc:504] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:32.036160 32452 leader_election.cc:288] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547), 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391)
I20260812 16:44:32.036954 31814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c61c39ddce4300b9a18dbebbeca1c9" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4432546da946e8851b4f9cd5499335" is_pre_election: true
I20260812 16:44:32.037279 31814 raft_consensus.cc:2454] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 0.
I20260812 16:44:32.037220 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c61c39ddce4300b9a18dbebbeca1c9" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4dd8917579442fb234097fc5c46d54" is_pre_election: true
I20260812 16:44:32.037495 32059 raft_consensus.cc:2454] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 0.
I20260812 16:44:32.037796 32197 leader_election.cc:302] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b998c59676b4507855ba921d0ad817b, ab4432546da946e8851b4f9cd5499335; no voters:
I20260812 16:44:32.038228 32452 raft_consensus.cc:2781] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:32.038326 32452 raft_consensus.cc:482] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:32.038414 32452 raft_consensus.cc:3037] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:32.040794 32452 raft_consensus.cc:504] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:32.041594 32452 leader_election.cc:288] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 election: Requested vote from peers ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547), 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391)
I20260812 16:44:32.042250 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c61c39ddce4300b9a18dbebbeca1c9" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4dd8917579442fb234097fc5c46d54"
I20260812 16:44:32.042250 31814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c61c39ddce4300b9a18dbebbeca1c9" candidate_uuid: "4b998c59676b4507855ba921d0ad817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4432546da946e8851b4f9cd5499335"
I20260812 16:44:32.042534 31814 raft_consensus.cc:3037] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:32.042533 32059 raft_consensus.cc:3037] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:32.047209 31814 raft_consensus.cc:2454] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 1.
I20260812 16:44:32.047209 32059 raft_consensus.cc:2454] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b998c59676b4507855ba921d0ad817b in term 1.
I20260812 16:44:32.049099 32198 leader_election.cc:302] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b4dd8917579442fb234097fc5c46d54, 4b998c59676b4507855ba921d0ad817b; no voters:
I20260812 16:44:32.049650 32452 raft_consensus.cc:2781] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:32.049866 32452 raft_consensus.cc:686] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [term 1 LEADER]: Becoming Leader. State: Replica: 4b998c59676b4507855ba921d0ad817b, State: Running, Role: LEADER
I20260812 16:44:32.050436 32452 consensus_queue.cc:237] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } }
I20260812 16:44:32.053712 32498 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: TxnStatusTablet state changed. Reason: New leader 4b998c59676b4507855ba921d0ad817b. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.054203 32498 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:44:32.055593 31516 catalog_manager.cc:5495] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b998c59676b4507855ba921d0ad817b (127.28.243.3). New cstate: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } health_report { overall_health: HEALTHY } } }
I20260812 16:44:32.057235 32516 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 16:44:32.057598 32516 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 16:44:32.095975 32498 consensus_queue.cc:1035] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:32.097177 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.084743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:39111 (local address 127.0.0.1:40736)
0812 16:44:32.085176 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:32.085211 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:32.085421 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:32.086367 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:32.086371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:32.086383 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:32.086572 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:32.086593 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.088854 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.088860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:32.090687 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:32.090713 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.093543 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.093556 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:32.093880 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.094025 (+ 145us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:32.096148 (+ 2123us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:32.096196 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 16:44:32.096283 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:44:32.096689 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 16:44:32.097553 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.084726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.28.243.3:39111)
0812 16:44:32.085825 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.085831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.085855 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:44:32.085967 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.085972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.086109 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.086324 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:32.086830 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.088628 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.092066 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.093093 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.094047 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.094504 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:32.095947 (+ 1443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:32.096924 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.097021 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.097141 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":666,"threads_started":1}
I20260812 16:44:32.108413 32059 raft_consensus.cc:1261] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Refusing update from remote peer 4b998c59676b4507855ba921d0ad817b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:32.113026 32452 consensus_queue.cc:1035] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:32.123085 32452 consensus_queue.cc:1035] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:32.124379 31814 raft_consensus.cc:1261] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Refusing update from remote peer 4b998c59676b4507855ba921d0ad817b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:32.127535 32530 consensus_queue.cc:1035] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:32.144121 32450 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: TxnStatusTablet state changed. Reason: New leader 4b998c59676b4507855ba921d0ad817b. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.144585 32450 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:32.151448 32529 mvcc.cc:205] Tried to move back new op lower bound from 7317721383336108032 to 7317721383125135360. Current Snapshot: MvccSnapshot[applied={T|T < 7317721383336108032}]
I20260812 16:44:32.152355 32450 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.152634 32450 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P ab4432546da946e8851b4f9cd5499335: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:32.154639 32498 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.155030 32498 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:44:32.159963 32474 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: TxnStatusTablet state changed. Reason: New leader 4b998c59676b4507855ba921d0ad817b. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.160333 32474 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:32.162173 32530 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.163246 32530 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 4b998c59676b4507855ba921d0ad817b: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:44:32.167128 32535 mvcc.cc:205] Tried to move back new op lower bound from 7317721383336108032 to 7317721383125135360. Current Snapshot: MvccSnapshot[applied={T|T < 7317721383336108032}]
I20260812 16:44:32.171397 32474 tablet_replica.cc:439] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b998c59676b4507855ba921d0ad817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab4432546da946e8851b4f9cd5499335" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33547 } } peers { permanent_uuid: "3b4dd8917579442fb234097fc5c46d54" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33391 } } peers { permanent_uuid: "4b998c59676b4507855ba921d0ad817b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39111 } } }
I20260812 16:44:32.171808 32474 tablet_replica.cc:442] T 69c61c39ddce4300b9a18dbebbeca1c9 P 3b4dd8917579442fb234097fc5c46d54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:32.185218 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.176500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.28.243.3:39111)
0812 16:44:32.176797 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.176804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.177295 (+ 491us) server_negotiation.cc:407] Connection header received
0812 16:44:32.177521 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.177527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.177626 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.177735 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:32.178399 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.179567 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.181435 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.182353 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.183047 (+ 694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.183642 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:32.184170 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:32.184335 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.184757 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.184866 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260812 16:44:32.211115 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.203215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:60233 (local address 127.28.243.3:39111)
0812 16:44:32.203443 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.203450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.204365 (+ 915us) server_negotiation.cc:407] Connection header received
0812 16:44:32.204569 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.204575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.204686 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.204814 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:32.205355 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.206570 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.208399 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.209168 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.209333 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.210012 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:32.210174 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:32.210277 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.210424 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.210526 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:44:32.211139 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.203401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:39111 (local address 127.28.243.3:60233)
0812 16:44:32.204230 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:32.204250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:32.204412 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:32.204876 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:32.204880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:32.204890 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:32.205186 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:32.205202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.206800 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.206805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:32.208205 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:32.208227 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.209407 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.209415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:32.209663 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.209758 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:32.210284 (+ 526us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:32.210296 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 16:44:32.210376 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 16:44:32.210539 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":3,"thread_start_us":447,"threads_started":1}
I20260812 16:44:32.310515 31516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:32.310796 31516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:32.314226 31516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f328b6c7c1e6418494ad34b2dc6ef7e4: Sending DeleteTablet for 3 replicas of tablet 72f059fa5873435885d15216b363324e
I20260812 16:44:32.315722 32285 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f059fa5873435885d15216b363324e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:32 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:40732
I20260812 16:44:32.315872 31778 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f059fa5873435885d15216b363324e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:32 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:33912
I20260812 16:44:32.317761 32030 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f059fa5873435885d15216b363324e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:32 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:45868
I20260812 16:44:32.320174 32555 tablet_replica.cc:330] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: stopping tablet replica
I20260812 16:44:32.320442 32555 raft_consensus.cc:2229] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:32.320928 32555 raft_consensus.cc:2258] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:32.326071 32555 ts_tablet_manager.cc:1905] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:32.331353 32559 tablet_replica.cc:330] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: stopping tablet replica
I20260812 16:44:32.331749 32559 raft_consensus.cc:2229] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:44:32.332672 32559 raft_consensus.cc:2258] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:32.334491 32559 ts_tablet_manager.cc:1905] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:32.340032 32560 tablet_replica.cc:330] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: stopping tablet replica
I20260812 16:44:32.340296 32560 raft_consensus.cc:2229] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:32.340560 32560 raft_consensus.cc:2258] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:32.343453 32560 ts_tablet_manager.cc:1905] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:32.348275 32559 ts_tablet_manager.cc:1918] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:32.351509 32555 ts_tablet_manager.cc:1918] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:32.351810 32555 log.cc:1196] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/72f059fa5873435885d15216b363324e
I20260812 16:44:32.352154 32560 ts_tablet_manager.cc:1918] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:32.352355 32560 log.cc:1196] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/72f059fa5873435885d15216b363324e
I20260812 16:44:32.352866 32560 ts_tablet_manager.cc:1939] T 72f059fa5873435885d15216b363324e P 3b4dd8917579442fb234097fc5c46d54: Deleting consensus metadata
I20260812 16:44:32.353500 32555 ts_tablet_manager.cc:1939] T 72f059fa5873435885d15216b363324e P ab4432546da946e8851b4f9cd5499335: Deleting consensus metadata
I20260812 16:44:32.354385 32559 log.cc:1196] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/72f059fa5873435885d15216b363324e
Aug 12 16:44:32 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:32.354753 31497 catalog_manager.cc:4841] TS 3b4dd8917579442fb234097fc5c46d54 (127.28.243.2:33391): tablet 72f059fa5873435885d15216b363324e (table test-table [id=bda432ef472047ea9faf2e18dcb31b23]) successfully deleted
I20260812 16:44:32.355652 32559 ts_tablet_manager.cc:1939] T 72f059fa5873435885d15216b363324e P 4b998c59676b4507855ba921d0ad817b: Deleting consensus metadata
I20260812 16:44:32.355926 31496 catalog_manager.cc:4841] TS ab4432546da946e8851b4f9cd5499335 (127.28.243.1:33547): tablet 72f059fa5873435885d15216b363324e (table test-table [id=bda432ef472047ea9faf2e18dcb31b23]) successfully deleted
I20260812 16:44:32.357197 31498 catalog_manager.cc:4841] TS 4b998c59676b4507855ba921d0ad817b (127.28.243.3:39111): tablet 72f059fa5873435885d15216b363324e (table test-table [id=bda432ef472047ea9faf2e18dcb31b23]) successfully deleted
I20260812 16:44:32.362990 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.325629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.28.243.1:33547)
0812 16:44:32.331497 (+ 5868us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.331505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.341917 (+ 10412us) server_negotiation.cc:407] Connection header received
0812 16:44:32.342169 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.342175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.342298 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.342464 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:32.345397 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.346571 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.348717 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.349420 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.349548 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.356135 (+ 6587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:32.356157 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:32.356182 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:32.356242 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:32.359816 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.360678 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.360687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.360689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.360845 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.361252 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.361256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.361258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.361683 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:32.361989 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.362318 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.362567 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5672,"thread_start_us":5488,"threads_started":1}
W20260812 16:44:32.363885 31883 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33918
I20260812 16:44:32.384852 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.367486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.28.243.62:35717)
0812 16:44:32.368616 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.368621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.368639 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:44:32.368707 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.368711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.368776 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.368880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:32.370415 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.371660 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.373013 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.373463 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.373588 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.376968 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:32.376977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:32.376982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:32.377038 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:32.381069 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.382255 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.382262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.382265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.382485 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.383073 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.383078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.383082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.383522 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:32.383785 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.384175 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.384457 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":773,"threads_started":1}
W20260812 16:44:32.385532 31516 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33602
I20260812 16:44:32.387888 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31650
I20260812 16:44:32.404105 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.389261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.28.243.1:33547)
0812 16:44:32.389475 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.389481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.390006 (+ 525us) server_negotiation.cc:407] Connection header received
0812 16:44:32.390195 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.390200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.390270 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.390355 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:32.391669 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.392452 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.393966 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.394419 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.394541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.397358 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:32.397363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:32.397366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:32.397411 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:32.400318 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.401239 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.401245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.401247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.401400 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.401827 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.401832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.401835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.402148 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:32.402404 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.403254 (+ 850us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.403610 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 16:44:32.404901 31883 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932
W20260812 16:44:32.405676 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:32.405907 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31650
I20260812 16:44:32.455163 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31899
Aug 12 16:44:32 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:44:32.473326 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.456072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.28.243.2:33391)
0812 16:44:32.456882 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.456886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.456903 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:44:32.456961 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.456964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.457026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.457149 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:32.458469 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.459362 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.460685 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.461132 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.461540 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.466854 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:32.466876 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:32.466895 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:32.466976 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:32.470011 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.471074 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.471082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.471084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.471259 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.471686 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.471689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.471691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.472099 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:32.472350 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.472776 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.473006 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":545,"threads_started":1}
W20260812 16:44:32.474004 32144 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45886
W20260812 16:44:32.474743 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:32.474982 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31899
I20260812 16:44:32.528668 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32164
Aug 12 16:44:32 dist-test-slave-1vmg krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:44:32.546234 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:32.529487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.28.243.3:39111)
0812 16:44:32.529678 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:32.529684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:32.529777 (+ 93us) server_negotiation.cc:407] Connection header received
0812 16:44:32.529972 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:32.529977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:32.530055 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:32.530151 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:32.531563 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.532359 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.533654 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:32.534189 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:32.534316 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:32.539152 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:32.539171 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:32.539188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:32.539240 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:32.542833 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.543929 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.543937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.543940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.544113 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:32.544517 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:32.544521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:32.544523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:32.544974 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:32.545316 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:32.545616 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 16:44:32.545890 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 16:44:32.546991 32400 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40756
W20260812 16:44:32.547772 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:32.547920 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 32164
I20260812 16:44:32.585747 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31468
I20260812 16:44:32.586988 31591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:32.712744 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31468
2026-08-12T16:44:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5882 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:32 dist-test-slave-1vmg krb5kdc[32579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:32 dist-test-slave-1vmg krb5kdc[32579](info): set up 2 sockets
Aug 12 16:44:32 dist-test-slave-1vmg krb5kdc[32579](info): commencing operation
krb5kdc: starting...
W20260812 16:44:34.016781 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:34 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:34.271668 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45937
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:45937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:34.631951 32603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:34.632292 32603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:34.632400 32603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:34.640321 32603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:34.640468 32603 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 16:44:34.640527 32603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:34.640625 32603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:34.640692 32603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:34.640780 32603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:34.651558 32603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:45937
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:34.653362 32603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:34.654945 32603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:34.667579 32611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:34.667651 32617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:34.667672 32612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:34.668859 32603 server_base.cc:1034] running on GCE node
I20260812 16:44:34.669688 32603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:34.671394 32603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:34.672674 32603 hybrid_clock.cc:648] HybridClock initialized: now 1786553074672582 us; error 52 us; skew 500 ppm
I20260812 16:44:34.673053 32603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:34 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:34.676664 32603 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:34.678620 32603 webserver.cc:460] Webserver started at http://127.28.243.62:35201/ using document root <none> and password file <none>
I20260812 16:44:34.679454 32603 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:34.679569 32603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:34.679926 32603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:34.683823 32603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "c1b53136acee45cb884d6ae850f2f601"
format_stamp: "Formatted at 2026-08-12 16:44:34 on dist-test-slave-1vmg"
server_key: "fb92140c6f2e78dc57ae0c627f887969"
server_key_iv: "8ed713e9f5aaef8ad1c03ae59711cae0"
server_key_version: "encryptionkey@0"
I20260812 16:44:34.684671 32603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "c1b53136acee45cb884d6ae850f2f601"
format_stamp: "Formatted at 2026-08-12 16:44:34 on dist-test-slave-1vmg"
server_key: "fb92140c6f2e78dc57ae0c627f887969"
server_key_iv: "8ed713e9f5aaef8ad1c03ae59711cae0"
server_key_version: "encryptionkey@0"
I20260812 16:44:34.691891 32603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:44:34.696877 32625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:34.698535 32603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:44:34.698762 32603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "c1b53136acee45cb884d6ae850f2f601"
format_stamp: "Formatted at 2026-08-12 16:44:34 on dist-test-slave-1vmg"
server_key: "fb92140c6f2e78dc57ae0c627f887969"
server_key_iv: "8ed713e9f5aaef8ad1c03ae59711cae0"
server_key_version: "encryptionkey@0"
I20260812 16:44:34.699065 32603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:34.724071 32603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:34.732219 32603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:34.732615 32603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:34.759989 32603 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:45937
I20260812 16:44:34.760062 32747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:45937 every 8 connection(s)
I20260812 16:44:34.761600 32603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:34.767797 32749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:34.768564 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 32603
I20260812 16:44:34.768772 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:34.769554 29644 external_mini_cluster.cc:1428] Setting key d1b83e26450452f67d84264855a25343
I20260812 16:44:34.782155 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Bootstrap starting.
I20260812 16:44:34.786454 32749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:34.787973 32749 log.cc:824] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:44:34 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:34.791761 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: No bootstrap required, opened a new log
I20260812 16:44:34.800864 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:34.773176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.28.243.62:45937)
0812 16:44:34.774489 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:34.774502 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:34.774556 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:44:34.776360 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:34.776387 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:34.776845 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:34.777215 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:34.779198 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:34.780868 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:34.783201 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:34.784705 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:34.784999 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:34.790494 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:34.790514 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:34.790536 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:34.790583 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:34.795203 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:34.796760 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:34.796770 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:34.796778 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:34.797006 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:34.797531 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:34.797536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:34.797538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:34.798238 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:34.798517 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:34.799128 (+ 611us) server_negotiation.cc:299] Negotiation successful
0812 16:44:34.799623 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":802,"threads_started":1}
I20260812 16:44:34.802807 32749 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b53136acee45cb884d6ae850f2f601" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45937 } }
I20260812 16:44:34.803393 32749 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:34.803505 32749 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b53136acee45cb884d6ae850f2f601, State: Initialized, Role: FOLLOWER
I20260812 16:44:34.804230 32749 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b53136acee45cb884d6ae850f2f601" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45937 } }
I20260812 16:44:34.804502 32749 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:34.804639 32749 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:34.804852 32749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:34.808004 32749 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b53136acee45cb884d6ae850f2f601" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45937 } }
I20260812 16:44:34.808596 32749 leader_election.cc:302] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1b53136acee45cb884d6ae850f2f601; no voters:
I20260812 16:44:34.809885 32749 leader_election.cc:288] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:34.811023 32756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:34.811815 32756 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [term 1 LEADER]: Becoming Leader. State: Replica: c1b53136acee45cb884d6ae850f2f601, State: Running, Role: LEADER
I20260812 16:44:34.812610 32756 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b53136acee45cb884d6ae850f2f601" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45937 } }
I20260812 16:44:34.812849 32749 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:34.815572 32761 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b53136acee45cb884d6ae850f2f601. Latest consensus state: current_term: 1 leader_uuid: "c1b53136acee45cb884d6ae850f2f601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b53136acee45cb884d6ae850f2f601" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45937 } } }
I20260812 16:44:34.815939 32761 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:34.816427 32760 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1b53136acee45cb884d6ae850f2f601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b53136acee45cb884d6ae850f2f601" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45937 } } }
I20260812 16:44:34.816753 32760 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:34.816991 303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:34.829068 303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:34.841610 303 catalog_manager.cc:1347] Generated new cluster ID: 69ef4983aa68410cb3000daba6be8b59
I20260812 16:44:34.841763 303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:34.864497 303 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:34.866372 303 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:34.873991 303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Generated new TSK 0
I20260812 16:44:34.874874 303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:34.944695 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45937
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:35.367609 325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:35.367944 325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:35.368064 325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:35.375727 325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:35.375867 325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:35.375957 325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 16:44:35.376012 325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:44:35.376111 325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:35.386615 325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:45937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:35.388101 325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:35.389513 325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:35.404305 339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:35.404316 342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:35.404305 348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:35.405894 325 server_base.cc:1034] running on GCE node
I20260812 16:44:35.406819 325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:35.408721 325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:35.410110 325 hybrid_clock.cc:648] HybridClock initialized: now 1786553075410020 us; error 64 us; skew 500 ppm
I20260812 16:44:35.410653 325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:35 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:35.415153 325 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:35.417227 325 webserver.cc:460] Webserver started at http://127.28.243.1:39707/ using document root <none> and password file <none>
I20260812 16:44:35.418018 325 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:35.418155 325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:35.418516 325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:35.421906 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
format_stamp: "Formatted at 2026-08-12 16:44:35 on dist-test-slave-1vmg"
server_key: "409a9adeb4d452d26d8046961495c40b"
server_key_iv: "293a557a726d333d15fe6c64602330d6"
server_key_version: "encryptionkey@0"
I20260812 16:44:35.422832 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
format_stamp: "Formatted at 2026-08-12 16:44:35 on dist-test-slave-1vmg"
server_key: "409a9adeb4d452d26d8046961495c40b"
server_key_iv: "293a557a726d333d15fe6c64602330d6"
server_key_version: "encryptionkey@0"
I20260812 16:44:35.429215 325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:44:35.434089 357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:35.435663 325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:44:35.435855 325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
format_stamp: "Formatted at 2026-08-12 16:44:35 on dist-test-slave-1vmg"
server_key: "409a9adeb4d452d26d8046961495c40b"
server_key_iv: "293a557a726d333d15fe6c64602330d6"
server_key_version: "encryptionkey@0"
I20260812 16:44:35.436070 325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:35.455524 325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:35.460640 325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:35.460974 325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:35.465814 325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:35.465947 325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:35.466076 325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:35.466168 325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:44:35 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:35.490687 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:35.464935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:33779 (local address 127.28.243.62:45937)
0812 16:44:35.465412 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:35.465418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:35.468052 (+ 2634us) server_negotiation.cc:407] Connection header received
0812 16:44:35.470012 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:35.470022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:35.470161 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:35.470498 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:35.472565 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.473497 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.476986 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.477542 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.477684 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:35.484041 (+ 6357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:35.484048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:35.484052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:35.484092 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:35.487338 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:35.488201 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:35.488207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:35.488211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:35.488343 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:35.488805 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:35.488811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:35.488814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:35.489101 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:35.489240 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:35.490065 (+ 825us) server_negotiation.cc:299] Negotiation successful
0812 16:44:35.490316 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260812 16:44:35.492274 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:35.466197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.28.243.1:33779)
0812 16:44:35.467297 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:35.467371 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:35.469556 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:35.470704 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:35.470713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:35.471411 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:35.472288 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:35.472315 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.473731 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.473738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:35.476736 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:35.476772 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.477830 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.477839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:35.478232 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:35.479208 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:35.479260 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:35.483655 (+ 4395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:35.487551 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:35.487561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:35.487582 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:35.488056 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:35.488457 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:35.488461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:35.488463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:35.488674 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:35.489252 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:35.489261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:35.489651 (+ 390us) client_negotiation.cc:769] Sending connection context
0812 16:44:35.489916 (+ 265us) client_negotiation.cc:240] Negotiation successful
0812 16:44:35.490511 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":392,"threads_started":1}
I20260812 16:44:35.519014 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:35.499239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:50475 (local address 127.28.243.62:45937)
0812 16:44:35.499481 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:35.499490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:35.500606 (+ 1116us) server_negotiation.cc:407] Connection header received
0812 16:44:35.500855 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:35.500861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:35.500967 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:35.501086 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:35.503699 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.504694 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.507603 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.508364 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.509471 (+ 1107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:35.511814 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:35.511821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:35.511825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:35.511861 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:35.515502 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:35.516550 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:35.516557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:35.516561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:35.516764 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:35.517308 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:35.517313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:35.517317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:35.517737 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:35.517910 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:35.518504 (+ 594us) server_negotiation.cc:299] Negotiation successful
0812 16:44:35.518711 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:44:35.519273 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:35.499206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.28.243.1:50475)
0812 16:44:35.500449 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:35.500481 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:35.500688 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:35.501292 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:35.501301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:35.502235 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:35.503305 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:35.503353 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.504936 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.504946 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:35.507170 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:35.507203 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:35.508767 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:35.508774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:35.508995 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:35.510344 (+ 1349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:35.510368 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:35.511551 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:35.515724 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:35.515731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:35.515739 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:35.516388 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:35.516917 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:35.516923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:35.516928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:35.517145 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:35.517859 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:35.517865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:35.518169 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 16:44:35.518374 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 16:44:35.518745 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":813,"threads_started":1}
I20260812 16:44:35.528234 567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:35209 every 8 connection(s)
I20260812 16:44:35.528236 325 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:35209
I20260812 16:44:35.530834 325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:35.534129 573 heartbeater.cc:347] Connected to a master server at 127.28.243.62:45937
I20260812 16:44:35.534621 573 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:35.535524 573 heartbeater.cc:511] Master 127.28.243.62:45937 requested a full tablet report, sending...
I20260812 16:44:35.536002 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 325
I20260812 16:44:35.536234 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:35.536994 29644 external_mini_cluster.cc:1428] Setting key 6ab0b0f49efe78f847aa6cbc3ebfee21
I20260812 16:44:35.538094 32658 ts_manager.cc:194] Registered new tserver with Master: c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209)
I20260812 16:44:35.540037 32658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:33779
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:35.596767 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45937
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:35.962545 578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:35.962890 578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:35.963055 578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:35.971885 578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:35.972030 578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:35.972168 578 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 16:44:35.972239 578 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:44:35.972355 578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:35.984623 578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:45937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:35.986249 578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:35.987856 578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:36.002853 587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:36.002947 586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:36.002988 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:36.003471 578 server_base.cc:1034] running on GCE node
I20260812 16:44:36.004772 578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:36.006074 578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:36.007318 578 hybrid_clock.cc:648] HybridClock initialized: now 1786553076007268 us; error 42 us; skew 500 ppm
I20260812 16:44:36.008081 578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:36 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:36.012363 578 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:36.014240 578 webserver.cc:460] Webserver started at http://127.28.243.2:45927/ using document root <none> and password file <none>
I20260812 16:44:36.015089 578 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:36.015221 578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:36.015578 578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:36.018986 578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "d355e2f44b824c258e73d8ea55fff0ff"
format_stamp: "Formatted at 2026-08-12 16:44:36 on dist-test-slave-1vmg"
server_key: "659b552c0b6a2886bb7fdfe07950a436"
server_key_iv: "6ba4d1f2484470d6910c39c72d1779a5"
server_key_version: "encryptionkey@0"
I20260812 16:44:36.019966 578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "d355e2f44b824c258e73d8ea55fff0ff"
format_stamp: "Formatted at 2026-08-12 16:44:36 on dist-test-slave-1vmg"
server_key: "659b552c0b6a2886bb7fdfe07950a436"
server_key_iv: "6ba4d1f2484470d6910c39c72d1779a5"
server_key_version: "encryptionkey@0"
I20260812 16:44:36.026654 578 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:44:36.031131 602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:36.033147 578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:44:36.033365 578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "d355e2f44b824c258e73d8ea55fff0ff"
format_stamp: "Formatted at 2026-08-12 16:44:36 on dist-test-slave-1vmg"
server_key: "659b552c0b6a2886bb7fdfe07950a436"
server_key_iv: "6ba4d1f2484470d6910c39c72d1779a5"
server_key_version: "encryptionkey@0"
I20260812 16:44:36.033594 578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:36.047693 578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:36.053970 578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:36.054314 578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:36.057500 578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:36.057668 578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:36.057799 578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:36.057874 578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:44:36 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:36.088901 616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.057866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:59813 (local address 127.28.243.62:45937)
0812 16:44:36.058439 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.058447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.059658 (+ 1211us) server_negotiation.cc:407] Connection header received
0812 16:44:36.061820 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.061826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.061915 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.062011 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:36.064199 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.066419 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.070153 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.070828 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.071040 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.077881 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:36.077891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:36.077896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:36.077953 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:36.081799 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.083542 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.083549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.083553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.084099 (+ 546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.085114 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.085121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.085124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.085583 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:36.085798 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.087818 (+ 2020us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.088159 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
I20260812 16:44:36.089980 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.058555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.28.243.2:59813)
0812 16:44:36.059471 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.059519 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.061501 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.062302 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.062311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.063090 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:36.063984 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.064010 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.066663 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.066673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.069860 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.069886 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.071458 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.071482 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.071835 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.073045 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:36.073085 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:36.077440 (+ 4355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:36.082061 (+ 4621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.082075 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.082105 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.083244 (+ 1139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.084373 (+ 1129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.084379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.084383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.084895 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.085785 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:36.085795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:36.086524 (+ 729us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.086953 (+ 429us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.087656 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":380,"threads_started":1}
I20260812 16:44:36.114831 578 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:36865
I20260812 16:44:36.115276 827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:36865 every 8 connection(s)
I20260812 16:44:36.116559 578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:36.119663 828 heartbeater.cc:347] Connected to a master server at 127.28.243.62:45937
I20260812 16:44:36.120324 828 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:36.120728 616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.100173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:37373 (local address 127.28.243.62:45937)
0812 16:44:36.100394 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.100400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.101346 (+ 946us) server_negotiation.cc:407] Connection header received
0812 16:44:36.101469 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.101476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.101576 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.101708 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:36.103205 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.104462 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.109262 (+ 4800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.109929 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.110105 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.113841 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:36.113853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:36.113859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:36.113959 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:36.117266 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.118339 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.118344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.118348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.118519 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.118954 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.118960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.118963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.119434 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:36.119606 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.120141 (+ 535us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.120383 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 16:44:36.120817 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.100301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.28.243.2:37373)
0812 16:44:36.101195 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.101213 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.101358 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.101761 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.101766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.102213 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:36.102813 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.102846 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.104795 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.104806 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.109010 (+ 4204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.109038 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.110425 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.110436 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.110666 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.111556 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:36.111584 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:36.113229 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:36.117492 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.117498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.117506 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.118125 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.118642 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.118648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.118652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.118796 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.119543 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:36.119548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:36.119783 (+ 235us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.119925 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.120335 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":452,"threads_started":1}
I20260812 16:44:36.121271 828 heartbeater.cc:511] Master 127.28.243.62:45937 requested a full tablet report, sending...
I20260812 16:44:36.123514 32658 ts_manager.cc:194] Registered new tserver with Master: d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:36.124872 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 578
I20260812 16:44:36.125070 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:36.125128 32658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:59813
I20260812 16:44:36.125689 29644 external_mini_cluster.cc:1428] Setting key 4fb17f06214002ac9155f5ca537a8e1c
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:36.188269 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45937
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:36.543520 573 heartbeater.cc:503] Master 127.28.243.62:45937 was elected leader, sending a full tablet report...
W20260812 16:44:36.569871 836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:36.570221 836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:36.570369 836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:36.577919 836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:36.578065 836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:36.578204 836 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 16:44:36.578279 836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:44:36.578389 836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:36.589020 836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:45937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:36.590482 836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:36.592037 836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:36.608129 849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:36.608089 847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:36.608172 852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:36.609105 836 server_base.cc:1034] running on GCE node
I20260812 16:44:36.609860 836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:36.611240 836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:36.612509 836 hybrid_clock.cc:648] HybridClock initialized: now 1786553076612448 us; error 46 us; skew 500 ppm
I20260812 16:44:36.612931 836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:36 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:36.617268 836 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:36.619611 836 webserver.cc:460] Webserver started at http://127.28.243.3:33865/ using document root <none> and password file <none>
I20260812 16:44:36.620638 836 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:36.620775 836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:36.621271 836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:36.625814 836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec"
format_stamp: "Formatted at 2026-08-12 16:44:36 on dist-test-slave-1vmg"
server_key: "0a9fa969c99148b245d96b43e0989202"
server_key_iv: "4d33b80d70a10799fae1717264a3b448"
server_key_version: "encryptionkey@0"
I20260812 16:44:36.627063 836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec"
format_stamp: "Formatted at 2026-08-12 16:44:36 on dist-test-slave-1vmg"
server_key: "0a9fa969c99148b245d96b43e0989202"
server_key_iv: "4d33b80d70a10799fae1717264a3b448"
server_key_version: "encryptionkey@0"
I20260812 16:44:36.635635 836 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260812 16:44:36.640499 860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:36.642638 836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260812 16:44:36.642868 836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec"
format_stamp: "Formatted at 2026-08-12 16:44:36 on dist-test-slave-1vmg"
server_key: "0a9fa969c99148b245d96b43e0989202"
server_key_iv: "4d33b80d70a10799fae1717264a3b448"
server_key_version: "encryptionkey@0"
I20260812 16:44:36.643169 836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:36.673925 836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:36.679189 836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:36.679492 836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:36.682528 836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:36.682649 836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:36.682771 836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:36.682873 836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:44:36 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553076, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:36.714113 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.683568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:44001 (local address 127.28.243.62:45937)
0812 16:44:36.684089 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.684096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.685360 (+ 1264us) server_negotiation.cc:407] Connection header received
0812 16:44:36.687678 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.687685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.687789 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.687901 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:36.690385 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.692180 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.694810 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.695514 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.695693 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.701404 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:36.701413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:36.701419 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:36.701471 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:36.706877 (+ 5406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.708525 (+ 1648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.708532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.708536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.708747 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.710126 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.710132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.710136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.710618 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:36.710784 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.713449 (+ 2665us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.713697 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
I20260812 16:44:36.715174 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.684224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.28.243.3:44001)
0812 16:44:36.685168 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.685228 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.687340 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.688289 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.688306 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.689146 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:36.690083 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.690124 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.692491 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.692496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.694544 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.694571 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.696113 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.696126 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.696484 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.697379 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:36.697415 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:36.701043 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:36.707402 (+ 6359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.707420 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.707443 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.708318 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.708926 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.709051 (+ 125us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.709058 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.709718 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.710955 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:36.710966 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:36.711703 (+ 737us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.712621 (+ 918us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.713459 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":402,"threads_started":1}
I20260812 16:44:36.733510 836 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:37837
I20260812 16:44:36.733620 1067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:37837 every 8 connection(s)
I20260812 16:44:36.735545 836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:36.738478 1071 heartbeater.cc:347] Connected to a master server at 127.28.243.62:45937
I20260812 16:44:36.738765 1071 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:36.739933 1071 heartbeater.cc:511] Master 127.28.243.62:45937 requested a full tablet report, sending...
I20260812 16:44:36.740437 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 836
I20260812 16:44:36.740643 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:36.741255 29644 external_mini_cluster.cc:1428] Setting key 20b58343e3bb62986ff34169cab2b828
I20260812 16:44:36.741545 32658 ts_manager.cc:194] Registered new tserver with Master: e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837)
I20260812 16:44:36.742177 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.723607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.28.243.3:59333)
0812 16:44:36.724955 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.725011 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.725294 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.726156 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.726164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.727271 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:36.727832 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.727858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.730240 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.730246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.732069 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.732089 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.733445 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.733452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.733652 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.734360 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:36.734382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:36.735270 (+ 888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:36.739216 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.739222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.739227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.739688 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.740174 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:36.740180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:36.740183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:36.740351 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:36.741077 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:36.741083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:36.741290 (+ 207us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.741429 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.741794 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":851,"threads_started":1}
I20260812 16:44:36.742733 32658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:44001
I20260812 16:44:36.743328 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.723518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:59333 (local address 127.28.243.62:45937)
0812 16:44:36.724436 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.724447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.725183 (+ 736us) server_negotiation.cc:407] Connection header received
0812 16:44:36.725471 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.725479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.725608 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.725725 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:36.728164 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.729465 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.732304 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.732961 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.733053 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.735510 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:36.735516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:36.735519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:36.735558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:36.739070 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.739834 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.739840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.739844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.740024 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.740512 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.740517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.740521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.740918 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:36.742580 (+ 1662us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.742710 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.742996 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601}
I20260812 16:44:36.744688 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:44:36 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:44:36 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:36.815883 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.793700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.28.243.62:45937)
0812 16:44:36.793869 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.793875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.794064 (+ 189us) server_negotiation.cc:407] Connection header received
0812 16:44:36.794187 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.794191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.794268 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.794359 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:36.796020 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.796763 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.798182 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.798632 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.798743 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.808339 (+ 9596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:36.808348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:36.808353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:36.808420 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:36.812187 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.813232 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.813238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.813241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.813415 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:36.813911 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:36.813915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:36.813918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:36.814271 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:36.814503 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.815296 (+ 793us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.815556 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:44:36.821650 32657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:44:36.830292 32657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:36.865749 1099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.847788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60110)
0812 16:44:36.856249 (+ 8461us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.856269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.856419 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.857630 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.857639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.857677 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:36.858208 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.858228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.860116 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.860121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.862301 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.862315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.864295 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.864305 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.864617 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.864668 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.864832 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.865335 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8195,"thread_start_us":8014,"threads_started":1}
I20260812 16:44:36.866711 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.848269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.28.243.1:35209)
0812 16:44:36.855626 (+ 7357us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.855636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.856642 (+ 1006us) server_negotiation.cc:407] Connection header received
0812 16:44:36.856794 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.856803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.857099 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.857326 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:36.858442 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.859973 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.862495 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.863723 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.864153 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.864531 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.866147 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.866415 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7144,"thread_start_us":8278,"threads_started":1}
I20260812 16:44:36.867249 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.847153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:37837 (local address 127.0.0.1:46428)
0812 16:44:36.856083 (+ 8930us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.856130 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.856391 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.857723 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.857728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.857759 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:36.858651 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.858665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.860519 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.860525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.862742 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.862757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.865172 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.865180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.865364 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.865403 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.865516 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.865669 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8582,"thread_start_us":8351,"threads_started":1}
I20260812 16:44:36.867367 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.856987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.28.243.2:36865)
0812 16:44:36.858223 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.858232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.858865 (+ 633us) server_negotiation.cc:407] Connection header received
0812 16:44:36.859089 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.859099 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.859366 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.859581 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:36.860457 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.861986 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.864494 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.865896 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.866011 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.866210 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.866869 (+ 659us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.867077 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":901,"threads_started":1}
I20260812 16:44:36.866739 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.847485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.28.243.3:37837)
0812 16:44:36.855276 (+ 7791us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:36.855288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:36.856860 (+ 1572us) server_negotiation.cc:407] Connection header received
0812 16:44:36.856969 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:36.856976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:36.857232 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:36.857416 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:36.858890 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.860360 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.862966 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.864503 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.864647 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.864920 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:36.866172 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0812 16:44:36.866385 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7510,"thread_start_us":7217,"threads_started":1}
I20260812 16:44:36.868079 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:36.857875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.0.0.1:35666)
0812 16:44:36.858741 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:36.858757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:36.858890 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:36.859591 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:36.859597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:36.859624 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:36.859964 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.859976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.862137 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.862143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.864262 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:36.864281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:36.866414 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:36.866419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:36.866611 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:36.866643 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:44:36.867583 (+ 940us) client_negotiation.cc:240] Negotiation successful
0812 16:44:36.867753 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":561,"threads_started":1}
I20260812 16:44:36.871214 451 tablet_service.cc:1467] Processing CreateTablet for tablet 697ab8c589b841d78a12300f9e6bb5b0 (DEFAULT_TABLE table=test-table [id=82874cd3185045a5bdb0d26ec5ae84bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:36.871212 948 tablet_service.cc:1467] Processing CreateTablet for tablet 697ab8c589b841d78a12300f9e6bb5b0 (DEFAULT_TABLE table=test-table [id=82874cd3185045a5bdb0d26ec5ae84bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:36.871973 694 tablet_service.cc:1467] Processing CreateTablet for tablet 697ab8c589b841d78a12300f9e6bb5b0 (DEFAULT_TABLE table=test-table [id=82874cd3185045a5bdb0d26ec5ae84bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:36.872970 948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697ab8c589b841d78a12300f9e6bb5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:36.873288 451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697ab8c589b841d78a12300f9e6bb5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:36.873405 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697ab8c589b841d78a12300f9e6bb5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:36.888041 1106 tablet_bootstrap.cc:492] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Bootstrap starting.
I20260812 16:44:36.890518 1109 tablet_bootstrap.cc:492] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Bootstrap starting.
I20260812 16:44:36.891953 1108 tablet_bootstrap.cc:492] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Bootstrap starting.
I20260812 16:44:36.893496 1106 tablet_bootstrap.cc:654] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:36.895539 1106 log.cc:824] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:36.896063 1108 tablet_bootstrap.cc:654] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:36.896617 1109 tablet_bootstrap.cc:654] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:36.897369 1108 log.cc:824] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:36.897864 1109 log.cc:824] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:36.899983 1106 tablet_bootstrap.cc:492] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: No bootstrap required, opened a new log
I20260812 16:44:36.900318 1106 ts_tablet_manager.cc:1397] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260812 16:44:36.900820 1109 tablet_bootstrap.cc:492] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: No bootstrap required, opened a new log
I20260812 16:44:36.901078 1109 ts_tablet_manager.cc:1397] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260812 16:44:36.901163 1108 tablet_bootstrap.cc:492] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: No bootstrap required, opened a new log
I20260812 16:44:36.901448 1108 ts_tablet_manager.cc:1397] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 16:44:36.909451 1109 raft_consensus.cc:348] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:36.909907 1109 raft_consensus.cc:374] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:36.910006 1109 raft_consensus.cc:729] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Initialized, Role: FOLLOWER
I20260812 16:44:36.910672 1109 consensus_queue.cc:260] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:36.911922 1071 heartbeater.cc:503] Master 127.28.243.62:45937 was elected leader, sending a full tablet report...
I20260812 16:44:36.912717 1109 ts_tablet_manager.cc:1428] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260812 16:44:36.913944 1108 raft_consensus.cc:348] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:36.914047 1106 raft_consensus.cc:348] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:36.914505 1108 raft_consensus.cc:374] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:36.914500 1106 raft_consensus.cc:374] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:36.914606 1106 raft_consensus.cc:729] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c880cc41f9804aa7bd285b0b16d91c2a, State: Initialized, Role: FOLLOWER
I20260812 16:44:36.914618 1108 raft_consensus.cc:729] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Initialized, Role: FOLLOWER
I20260812 16:44:36.915310 1106 consensus_queue.cc:260] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:36.915311 1108 consensus_queue.cc:260] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:36.917026 828 heartbeater.cc:503] Master 127.28.243.62:45937 was elected leader, sending a full tablet report...
I20260812 16:44:36.917580 1106 ts_tablet_manager.cc:1428] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260812 16:44:36.918503 1108 ts_tablet_manager.cc:1428] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260812 16:44:36.987957 1072 tablet.cc:2369] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:44:37.034704 574 tablet.cc:2369] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:44:37.119141 829 tablet.cc:2369] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:44:37.219822 1115 raft_consensus.cc:482] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:37.220291 1115 raft_consensus.cc:504] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:37.222505 1115 leader_election.cc:288] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837), d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:37.231365 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.223280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:46823 (local address 127.28.243.3:37837)
0812 16:44:37.223521 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.223527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.223964 (+ 437us) server_negotiation.cc:407] Connection header received
0812 16:44:37.224298 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.224306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.224407 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.224483 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:37.225722 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.226781 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.229186 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.230181 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.230473 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.230578 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.230859 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.231089 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 16:44:37.231423 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.222876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:37837 (local address 127.28.243.1:46823)
0812 16:44:37.223788 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.223812 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.224045 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.224751 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.224756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.224787 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:37.225456 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.225474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.227019 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.227025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.228940 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.228956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.230452 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.230459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.230681 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.230716 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.230799 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.230910 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":33,"thread_start_us":480,"threads_started":1}
I20260812 16:44:37.231410 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.224160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:59081 (local address 127.28.243.2:36865)
0812 16:44:37.224408 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.224414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.225156 (+ 742us) server_negotiation.cc:407] Connection header received
0812 16:44:37.225405 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.225409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.225478 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.225555 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:37.226154 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.227205 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.229482 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.230349 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.230539 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.230620 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.231023 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.231171 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 16:44:37.231731 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.223887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.28.243.1:59081)
0812 16:44:37.225012 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.225031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.225249 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.225608 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.225614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.225642 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:37.225950 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.225963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.227375 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.227379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.229270 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.229289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.230661 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.230667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.230826 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.230860 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.230991 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.231135 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":34,"spinlock_wait_cycles":32128,"thread_start_us":699,"threads_started":1}
I20260812 16:44:37.232625 985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697ab8c589b841d78a12300f9e6bb5b0" candidate_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" is_pre_election: true
I20260812 16:44:37.233120 985 raft_consensus.cc:2454] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c880cc41f9804aa7bd285b0b16d91c2a in term 0.
I20260812 16:44:37.233265 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697ab8c589b841d78a12300f9e6bb5b0" candidate_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d355e2f44b824c258e73d8ea55fff0ff" is_pre_election: true
I20260812 16:44:37.234087 734 raft_consensus.cc:2454] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c880cc41f9804aa7bd285b0b16d91c2a in term 0.
I20260812 16:44:37.234027 365 leader_election.cc:302] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c880cc41f9804aa7bd285b0b16d91c2a, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:37.234642 1115 raft_consensus.cc:2781] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:37.234826 1115 raft_consensus.cc:482] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:37.234978 1115 raft_consensus.cc:3037] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:37.237367 1115 raft_consensus.cc:504] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:37.238101 1115 leader_election.cc:288] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [CANDIDATE]: Term 1 election: Requested vote from peers e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837), d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:37.238706 985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697ab8c589b841d78a12300f9e6bb5b0" candidate_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec"
I20260812 16:44:37.238860 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697ab8c589b841d78a12300f9e6bb5b0" candidate_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d355e2f44b824c258e73d8ea55fff0ff"
I20260812 16:44:37.238989 985 raft_consensus.cc:3037] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:37.239154 734 raft_consensus.cc:3037] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:37.241667 734 raft_consensus.cc:2454] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c880cc41f9804aa7bd285b0b16d91c2a in term 1.
I20260812 16:44:37.241679 985 raft_consensus.cc:2454] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c880cc41f9804aa7bd285b0b16d91c2a in term 1.
I20260812 16:44:37.242429 365 leader_election.cc:302] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c880cc41f9804aa7bd285b0b16d91c2a, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:37.243146 1115 raft_consensus.cc:2781] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:37.243691 1115 raft_consensus.cc:686] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 LEADER]: Becoming Leader. State: Replica: c880cc41f9804aa7bd285b0b16d91c2a, State: Running, Role: LEADER
I20260812 16:44:37.244302 1115 consensus_queue.cc:237] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } }
I20260812 16:44:37.250077 32658 catalog_manager.cc:5495] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1). New cstate: current_term: 1 leader_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } health_report { overall_health: HEALTHY } } }
Aug 12 16:44:37 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:37.359433 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.341851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.28.243.62:45937)
0812 16:44:37.342651 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.342658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.342817 (+ 159us) server_negotiation.cc:407] Connection header received
0812 16:44:37.343021 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.343027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.343088 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.343180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:37.344431 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.345254 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.346681 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.347514 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.347726 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.352398 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:37.352405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:37.352409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:37.352451 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:37.355798 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:37.356622 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:37.356627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:37.356630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:37.356782 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:37.357388 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:37.357393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:37.357396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:37.357827 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:37.357995 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.358489 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.358903 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":622,"thread_start_us":408,"threads_started":1}
I20260812 16:44:37.359468 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.341864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45937 (local address 127.0.0.1:33706)
0812 16:44:37.342718 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.342734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.342892 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.343218 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.343222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.343974 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:37.344267 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.344277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.345389 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.345394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.346514 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.346528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.347944 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.347958 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.348138 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.348859 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:37.348880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:37.352050 (+ 3170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:37.355987 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:37.356004 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:37.356011 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:37.356500 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:37.356888 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:37.356891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:37.356894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:37.357209 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:37.357995 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:37.358008 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:37.358192 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.358359 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.358744 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":449,"threads_started":1}
I20260812 16:44:37.366393 32658 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:33706:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 16:44:37.368022 32658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:37.376914 694 tablet_service.cc:1467] Processing CreateTablet for tablet 4a63662c1f814ab6919b6d18e5e67eb6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=968d70987c4340e483442fe163143ff0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:44:37.376767 451 tablet_service.cc:1467] Processing CreateTablet for tablet 4a63662c1f814ab6919b6d18e5e67eb6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=968d70987c4340e483442fe163143ff0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:44:37.377009 948 tablet_service.cc:1467] Processing CreateTablet for tablet 4a63662c1f814ab6919b6d18e5e67eb6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=968d70987c4340e483442fe163143ff0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:44:37.377772 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a63662c1f814ab6919b6d18e5e67eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:37.377784 948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a63662c1f814ab6919b6d18e5e67eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:37.377781 451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a63662c1f814ab6919b6d18e5e67eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:37.383630 1108 tablet_bootstrap.cc:492] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: Bootstrap starting.
I20260812 16:44:37.385784 1108 tablet_bootstrap.cc:654] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:37.386161 1106 tablet_bootstrap.cc:492] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: Bootstrap starting.
I20260812 16:44:37.386212 1109 tablet_bootstrap.cc:492] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: Bootstrap starting.
I20260812 16:44:37.388135 1109 tablet_bootstrap.cc:654] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:37.389240 1106 tablet_bootstrap.cc:654] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:37.389986 1108 tablet_bootstrap.cc:492] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: No bootstrap required, opened a new log
I20260812 16:44:37.390177 1108 ts_tablet_manager.cc:1397] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260812 16:44:37.390969 1108 raft_consensus.cc:348] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.391265 1108 raft_consensus.cc:374] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:37.391376 1108 raft_consensus.cc:729] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Initialized, Role: FOLLOWER
I20260812 16:44:37.391652 1109 tablet_bootstrap.cc:492] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: No bootstrap required, opened a new log
I20260812 16:44:37.391755 1106 tablet_bootstrap.cc:492] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: No bootstrap required, opened a new log
I20260812 16:44:37.391928 1106 ts_tablet_manager.cc:1397] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 16:44:37.392107 1109 ts_tablet_manager.cc:1397] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 16:44:37.391629 1108 consensus_queue.cc:260] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.392791 1114 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.393023 1106 raft_consensus.cc:348] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.393203 1114 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.393272 1106 raft_consensus.cc:374] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:37.393239 1109 raft_consensus.cc:348] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.393383 1106 raft_consensus.cc:729] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c880cc41f9804aa7bd285b0b16d91c2a, State: Initialized, Role: FOLLOWER
I20260812 16:44:37.393483 1109 raft_consensus.cc:374] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:37.393595 1109 raft_consensus.cc:729] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Initialized, Role: FOLLOWER
I20260812 16:44:37.393627 1106 consensus_queue.cc:260] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.393862 1109 consensus_queue.cc:260] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.394178 1115 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.394395 1106 ts_tablet_manager.cc:1428] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 16:44:37.394495 1115 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.394341 1113 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.394672 1113 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.396067 1109 ts_tablet_manager.cc:1428] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260812 16:44:37.396132 1108 ts_tablet_manager.cc:1428] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260812 16:44:37.591766 1114 raft_consensus.cc:482] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:37.592036 1114 raft_consensus.cc:504] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.593740 1114 leader_election.cc:288] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837)
I20260812 16:44:37.600842 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.594310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:48701 (local address 127.28.243.3:37837)
0812 16:44:37.594520 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.594528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.595145 (+ 617us) server_negotiation.cc:407] Connection header received
0812 16:44:37.595447 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.595452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.595522 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.595603 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:37.596290 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.597032 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.598934 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.599824 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.599988 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.600095 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.600405 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.600548 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:44:37.600903 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.594109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:52757 (local address 127.28.243.1:35209)
0812 16:44:37.595306 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.595311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.595663 (+ 352us) server_negotiation.cc:407] Connection header received
0812 16:44:37.595820 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.595823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.595902 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.595988 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:37.596559 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.597282 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.599020 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.599881 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.600040 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.600115 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.600447 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.600588 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":905,"threads_started":1}
I20260812 16:44:37.600999 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.594095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:37837 (local address 127.28.243.2:48701)
0812 16:44:37.595023 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.595046 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.595285 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.595644 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.595649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.595750 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:37.596127 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.596138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.597180 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.597186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.598741 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.598754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.600096 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.600102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.600249 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.600277 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.600353 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.600496 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":64,"spinlock_wait_cycles":27392,"thread_start_us":1132,"threads_started":1}
I20260812 16:44:37.601399 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.594117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.28.243.2:52757)
0812 16:44:37.595516 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.595532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.595681 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.596035 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.596043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.596070 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:37.596415 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.596425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.597418 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.597421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.598847 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.598860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.600153 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.600160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.600291 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.600315 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.600405 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.600553 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":6,"thread_start_us":552,"threads_started":1}
I20260812 16:44:37.601761 985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a63662c1f814ab6919b6d18e5e67eb6" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" is_pre_election: true
I20260812 16:44:37.602013 985 raft_consensus.cc:2454] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 0.
I20260812 16:44:37.602253 496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a63662c1f814ab6919b6d18e5e67eb6" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" is_pre_election: true
I20260812 16:44:37.602479 609 leader_election.cc:302] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:37.602718 496 raft_consensus.cc:2454] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 0.
I20260812 16:44:37.602942 1114 raft_consensus.cc:2781] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:37.603066 1114 raft_consensus.cc:482] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:37.603130 1114 raft_consensus.cc:3037] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:37.606227 1114 raft_consensus.cc:504] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.606859 1114 leader_election.cc:288] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 election: Requested vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837)
I20260812 16:44:37.607580 985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a63662c1f814ab6919b6d18e5e67eb6" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec"
I20260812 16:44:37.607663 496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a63662c1f814ab6919b6d18e5e67eb6" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
I20260812 16:44:37.607857 496 raft_consensus.cc:3037] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:37.607872 985 raft_consensus.cc:3037] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:37.610982 985 raft_consensus.cc:2454] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 1.
I20260812 16:44:37.611008 496 raft_consensus.cc:2454] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 1.
I20260812 16:44:37.611493 603 leader_election.cc:302] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c880cc41f9804aa7bd285b0b16d91c2a, d355e2f44b824c258e73d8ea55fff0ff; no voters:
I20260812 16:44:37.611820 1114 raft_consensus.cc:2781] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:37.612294 1114 raft_consensus.cc:686] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [term 1 LEADER]: Becoming Leader. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Running, Role: LEADER
I20260812 16:44:37.612782 1114 consensus_queue.cc:237] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.616194 1141 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: TxnStatusTablet state changed. Reason: New leader d355e2f44b824c258e73d8ea55fff0ff. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.616604 1141 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:44:37.617391 1143 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 16:44:37.617695 1143 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 16:44:37.617668 32655 catalog_manager.cc:5495] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff reported cstate change: term changed from 0 to 1, leader changed from <none> to d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2). New cstate: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20260812 16:44:37.651732 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.643445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.0.0.1:35682)
0812 16:44:37.643708 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.643726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.643889 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.644684 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.644690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.644706 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:37.644959 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.644970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.646218 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.646224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.647589 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.647611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.649093 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.649102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.649323 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.649416 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.651018 (+ 1602us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.651051 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.651156 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.651385 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 16:44:37.651775 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.643727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.28.243.2:36865)
0812 16:44:37.644087 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.644109 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.644145 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:44:37.644307 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.644311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.644453 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.644624 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:37.645121 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.646067 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.647945 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.648765 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.648932 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.649794 (+ 862us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.650835 (+ 1041us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.651062 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.651194 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.651315 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 16:44:37.660353 985 raft_consensus.cc:1261] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Refusing update from remote peer d355e2f44b824c258e73d8ea55fff0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:37.660311 496 raft_consensus.cc:1261] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Refusing update from remote peer d355e2f44b824c258e73d8ea55fff0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:37.661402 1141 consensus_queue.cc:1035] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:37.661867 1114 consensus_queue.cc:1035] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:37.676077 1113 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: TxnStatusTablet state changed. Reason: New leader d355e2f44b824c258e73d8ea55fff0ff. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.676481 1113 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.678409 1115 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: TxnStatusTablet state changed. Reason: New leader d355e2f44b824c258e73d8ea55fff0ff. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.678820 1115 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.681576 1146 mvcc.cc:205] Tried to move back new op lower bound from 7317721406086131712 to 7317721405905735680. Current Snapshot: MvccSnapshot[applied={T|T < 7317721406086131712}]
I20260812 16:44:37.684494 1148 mvcc.cc:205] Tried to move back new op lower bound from 7317721406086131712 to 7317721405905735680. Current Snapshot: MvccSnapshot[applied={T|T < 7317721406086131712}]
I20260812 16:44:37.685402 1147 mvcc.cc:205] Tried to move back new op lower bound from 7317721406086131712 to 7317721405905735680. Current Snapshot: MvccSnapshot[applied={T|T < 7317721406086131712}]
I20260812 16:44:37.686136 1141 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.686846 1141 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:44:37.687716 1113 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.688253 1113 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P e8b5a0fa71ae4b06b7358d092a3f20ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.691133 1115 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.691519 1115 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P c880cc41f9804aa7bd285b0b16d91c2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:44:37.691290 1141 tablet_replica.cc:439] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } } }
I20260812 16:44:37.692782 1141 tablet_replica.cc:442] T 4a63662c1f814ab6919b6d18e5e67eb6 P d355e2f44b824c258e73d8ea55fff0ff: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:44:37.711861 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.704636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.28.243.1:35209)
0812 16:44:37.704956 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.704964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.705640 (+ 676us) server_negotiation.cc:407] Connection header received
0812 16:44:37.705941 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.705947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.706052 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.706229 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:37.706767 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.707597 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.709403 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.709877 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.710010 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.710506 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.711209 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.711354 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.711457 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.711568 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 16:44:37.734368 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.725936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44951 (local address 127.28.243.2:36865)
0812 16:44:37.726169 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.726175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.726747 (+ 572us) server_negotiation.cc:407] Connection header received
0812 16:44:37.727015 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.727022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.727162 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.727301 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:37.727964 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.729149 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.731073 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.731921 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.733324 (+ 1403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.733503 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.733644 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.733731 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.733900 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.734003 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:44:37.734378 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.725614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.28.243.1:44951)
0812 16:44:37.726570 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.726597 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.726818 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.727376 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.727381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.727395 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:37.727780 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.727790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.729374 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.729379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.730800 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.730819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.732432 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.732441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.732849 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.733164 (+ 315us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.733764 (+ 600us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.733779 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.733853 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.734057 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":553,"threads_started":1}
I20260812 16:44:37.749929 734 raft_consensus.cc:1261] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Refusing update from remote peer c880cc41f9804aa7bd285b0b16d91c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:37.751003 1115 consensus_queue.cc:1035] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 16:44:37.756337 985 raft_consensus.cc:1261] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Refusing update from remote peer c880cc41f9804aa7bd285b0b16d91c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:37.757172 1115 consensus_queue.cc:1035] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:37.821241 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.814209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:35209 (local address 127.28.243.1:35209)
0812 16:44:37.814437 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:37.814444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:37.814695 (+ 251us) server_negotiation.cc:407] Connection header received
0812 16:44:37.815023 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:37.815029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:37.815143 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:37.815216 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:37.816097 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.817291 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.819132 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.819644 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.819754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.820321 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.820521 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.820607 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:37.820801 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 16:44:37.820890 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:44:37.822108 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:37.813640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.28.243.2:35209)
0812 16:44:37.814565 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:37.814589 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:37.814788 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:37.815438 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:37.815442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:37.815455 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:37.815918 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.815933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.817426 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.817432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.818993 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:37.819009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:37.819846 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:37.819854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:37.820033 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:37.820113 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:37.820659 (+ 546us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:37.820671 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 16:44:37.821497 (+ 826us) client_negotiation.cc:240] Negotiation successful
0812 16:44:37.821662 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":1153,"threads_started":1}
I20260812 16:44:37.872053 32655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:37.872395 32655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:37.876689 32655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Sending DeleteTablet for 3 replicas of tablet 697ab8c589b841d78a12300f9e6bb5b0
I20260812 16:44:37.878408 948 tablet_service.cc:1514] Processing DeleteTablet for tablet 697ab8c589b841d78a12300f9e6bb5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:37 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260812 16:44:37.878412 694 tablet_service.cc:1514] Processing DeleteTablet for tablet 697ab8c589b841d78a12300f9e6bb5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:37 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:35666
I20260812 16:44:37.879287 451 tablet_service.cc:1514] Processing DeleteTablet for tablet 697ab8c589b841d78a12300f9e6bb5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:37 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60110
I20260812 16:44:37.891785 1184 tablet_replica.cc:330] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: stopping tablet replica
I20260812 16:44:37.890625 32655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 16:44:37.892441 1184 raft_consensus.cc:2229] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 16:44:37.892563 32655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:37.892855 1184 raft_consensus.cc:2258] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:37.895243 1184 ts_tablet_manager.cc:1905] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:37.903822 1184 ts_tablet_manager.cc:1918] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:37.904367 1184 log.cc:1196] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/697ab8c589b841d78a12300f9e6bb5b0
I20260812 16:44:37.905185 1184 ts_tablet_manager.cc:1939] T 697ab8c589b841d78a12300f9e6bb5b0 P d355e2f44b824c258e73d8ea55fff0ff: Deleting consensus metadata
I20260812 16:44:37.907156 694 tablet_service.cc:1467] Processing CreateTablet for tablet fe7c93f4acf1463b8428e44609a95165 (DEFAULT_TABLE table=test-table [id=bcefea7eaa7242a69f9971bc44ad5f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:37.907305 947 tablet_service.cc:1467] Processing CreateTablet for tablet fe7c93f4acf1463b8428e44609a95165 (DEFAULT_TABLE table=test-table [id=bcefea7eaa7242a69f9971bc44ad5f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:37.907773 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7c93f4acf1463b8428e44609a95165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:37.908100 947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7c93f4acf1463b8428e44609a95165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:37.908505 32630 catalog_manager.cc:4841] TS d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865): tablet 697ab8c589b841d78a12300f9e6bb5b0 (table test-table [id=82874cd3185045a5bdb0d26ec5ae84bf]) successfully deleted
I20260812 16:44:37.912070 449 tablet_service.cc:1467] Processing CreateTablet for tablet fe7c93f4acf1463b8428e44609a95165 (DEFAULT_TABLE table=test-table [id=bcefea7eaa7242a69f9971bc44ad5f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:37.912688 449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7c93f4acf1463b8428e44609a95165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:37.931180 1185 tablet_replica.cc:330] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: stopping tablet replica
I20260812 16:44:37.931924 1185 raft_consensus.cc:2229] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:44:37.933040 1185 raft_consensus.cc:2258] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:37.935926 1182 tablet_replica.cc:330] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: stopping tablet replica
I20260812 16:44:37.936497 1182 raft_consensus.cc:2229] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:37.936929 1182 raft_consensus.cc:2258] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:37.941123 1187 tablet_bootstrap.cc:492] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Bootstrap starting.
I20260812 16:44:37.942092 1185 ts_tablet_manager.cc:1905] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:37.940570 1182 ts_tablet_manager.cc:1905] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:37.953883 1188 tablet_bootstrap.cc:492] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Bootstrap starting.
I20260812 16:44:37.956110 1188 tablet_bootstrap.cc:654] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:37.956292 1190 tablet_bootstrap.cc:492] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Bootstrap starting.
I20260812 16:44:37.958302 1182 ts_tablet_manager.cc:1918] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:37.958513 1182 log.cc:1196] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/697ab8c589b841d78a12300f9e6bb5b0
I20260812 16:44:37.959071 1187 tablet_bootstrap.cc:654] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:37.959107 1182 ts_tablet_manager.cc:1939] T 697ab8c589b841d78a12300f9e6bb5b0 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting consensus metadata
I20260812 16:44:37.961201 32630 catalog_manager.cc:4841] TS e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837): tablet 697ab8c589b841d78a12300f9e6bb5b0 (table test-table [id=82874cd3185045a5bdb0d26ec5ae84bf]) successfully deleted
I20260812 16:44:37.958412 1190 tablet_bootstrap.cc:654] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:37.967644 1187 tablet_bootstrap.cc:492] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: No bootstrap required, opened a new log
I20260812 16:44:37.967975 1187 ts_tablet_manager.cc:1397] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.002s
I20260812 16:44:37.968016 1185 ts_tablet_manager.cc:1918] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:37.968290 1185 log.cc:1196] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/697ab8c589b841d78a12300f9e6bb5b0
I20260812 16:44:37.968910 1185 ts_tablet_manager.cc:1939] T 697ab8c589b841d78a12300f9e6bb5b0 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting consensus metadata
I20260812 16:44:37.970858 32626 catalog_manager.cc:4841] TS c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209): tablet 697ab8c589b841d78a12300f9e6bb5b0 (table test-table [id=82874cd3185045a5bdb0d26ec5ae84bf]) successfully deleted
I20260812 16:44:37.972527 1187 raft_consensus.cc:348] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.972857 1187 raft_consensus.cc:374] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:37.972958 1187 raft_consensus.cc:729] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Initialized, Role: FOLLOWER
I20260812 16:44:37.973243 1187 consensus_queue.cc:260] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.977667 1187 ts_tablet_manager.cc:1428] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260812 16:44:37.978686 1190 tablet_bootstrap.cc:492] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: No bootstrap required, opened a new log
I20260812 16:44:37.978863 1190 ts_tablet_manager.cc:1397] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.005s
I20260812 16:44:37.979830 1190 raft_consensus.cc:348] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.980104 1190 raft_consensus.cc:374] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:37.980216 1190 raft_consensus.cc:729] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c880cc41f9804aa7bd285b0b16d91c2a, State: Initialized, Role: FOLLOWER
I20260812 16:44:37.980526 1190 consensus_queue.cc:260] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.981169 1190 ts_tablet_manager.cc:1428] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 16:44:37.982890 1188 tablet_bootstrap.cc:492] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: No bootstrap required, opened a new log
I20260812 16:44:37.983174 1188 ts_tablet_manager.cc:1397] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.004s
I20260812 16:44:37.984056 1188 raft_consensus.cc:348] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.984422 1188 raft_consensus.cc:374] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:37.984669 1188 raft_consensus.cc:729] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Initialized, Role: FOLLOWER
I20260812 16:44:37.985102 1188 consensus_queue.cc:260] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:37.987543 1188 ts_tablet_manager.cc:1428] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260812 16:44:38.046768 1113 raft_consensus.cc:482] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:38.047276 1113 raft_consensus.cc:504] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:38.052143 1113 leader_election.cc:288] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:38.074410 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.051897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:39959 (local address 127.28.243.2:36865)
0812 16:44:38.052158 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:38.052164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:38.062561 (+ 10397us) server_negotiation.cc:407] Connection header received
0812 16:44:38.064034 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:38.064041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:38.064134 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:38.064204 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:38.065029 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.067069 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.070821 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.072769 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.072997 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.073103 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:38.073833 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 16:44:38.074030 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:44:38.075095 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.051553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:36169 (local address 127.28.243.1:35209)
0812 16:44:38.051816 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:38.051823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:38.064274 (+ 12451us) server_negotiation.cc:407] Connection header received
0812 16:44:38.064605 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:38.064611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:38.064702 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:38.064773 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:44:38.066537 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.068019 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.071707 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.073205 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.073524 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.073635 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:38.074487 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 16:44:38.074684 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 16:44:38.075555 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.050645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.28.243.3:39959)
0812 16:44:38.062407 (+ 11762us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:38.062436 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:38.063529 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:38.064430 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:38.064435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:38.064464 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:38.064835 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.064849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.067438 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.067445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.070463 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.070491 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.073351 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.073359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.073561 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.073601 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 16:44:38.074885 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0812 16:44:38.075160 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11582,"thread_start_us":13068,"threads_started":1}
I20260812 16:44:38.076543 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.050755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.28.243.3:36169)
0812 16:44:38.064137 (+ 13382us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:38.064160 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:38.064392 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:38.064990 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:38.064995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:38.065022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:44:38.066141 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.066180 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.068295 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.068397 (+ 102us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.071410 (+ 3013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.071431 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.074012 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.074020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.074253 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.074291 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:44:38.075968 (+ 1677us) client_negotiation.cc:240] Negotiation successful
0812 16:44:38.076142 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13204,"mutex_wait_us":5,"thread_start_us":12593,"threads_started":1}
I20260812 16:44:38.076480 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c93f4acf1463b8428e44609a95165" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d355e2f44b824c258e73d8ea55fff0ff" is_pre_election: true
I20260812 16:44:38.076808 734 raft_consensus.cc:2454] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 0.
I20260812 16:44:38.077489 865 leader_election.cc:302] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:38.077581 496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c93f4acf1463b8428e44609a95165" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" is_pre_election: true
I20260812 16:44:38.077934 496 raft_consensus.cc:2454] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 0.
I20260812 16:44:38.078006 1113 raft_consensus.cc:2781] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:38.078141 1113 raft_consensus.cc:482] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:38.078231 1113 raft_consensus.cc:3037] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:38.081663 1113 raft_consensus.cc:504] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:38.082520 1113 leader_election.cc:288] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 election: Requested vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:38.083307 496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c93f4acf1463b8428e44609a95165" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
I20260812 16:44:38.083357 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7c93f4acf1463b8428e44609a95165" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d355e2f44b824c258e73d8ea55fff0ff"
I20260812 16:44:38.083570 496 raft_consensus.cc:3037] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:38.083606 734 raft_consensus.cc:3037] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:38.088737 496 raft_consensus.cc:2454] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 1.
I20260812 16:44:38.088814 734 raft_consensus.cc:2454] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 1.
I20260812 16:44:38.089780 865 leader_election.cc:302] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:38.090660 1113 raft_consensus.cc:2781] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:38.091698 1113 raft_consensus.cc:686] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 LEADER]: Becoming Leader. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Running, Role: LEADER
I20260812 16:44:38.092760 1113 consensus_queue.cc:237] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:38.097918 32655 catalog_manager.cc:5495] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec reported cstate change: term changed from 0 to 1, leader changed from <none> to e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3). New cstate: current_term: 1 leader_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } health_report { overall_health: HEALTHY } } }
I20260812 16:44:38.177590 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.156741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.28.243.3:37837)
0812 16:44:38.168866 (+ 12125us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:38.168874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:38.168915 (+ 41us) server_negotiation.cc:407] Connection header received
0812 16:44:38.169094 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:38.169101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:38.169254 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:38.169588 (+ 334us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:38.170431 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.171887 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.174203 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.175009 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.175225 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.175975 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:38.176778 (+ 803us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:38.176940 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:38.177101 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 16:44:38.177224 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11897,"thread_start_us":11694,"threads_started":1}
I20260812 16:44:38.208462 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.197274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.28.243.3:55415)
0812 16:44:38.198067 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:38.198090 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:38.198362 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:38.198822 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:38.198827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:38.198841 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:38.199336 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.199349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.200917 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.200923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.202649 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.202671 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.205929 (+ 3258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.205959 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.206376 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.206551 (+ 175us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:38.207623 (+ 1072us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:38.207645 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 16:44:38.207816 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 16:44:38.208029 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":351,"threads_started":1}
I20260812 16:44:38.208492 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.197148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:55415 (local address 127.28.243.2:36865)
0812 16:44:38.197390 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:38.197396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:38.198208 (+ 812us) server_negotiation.cc:407] Connection header received
0812 16:44:38.198551 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:38.198557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:38.198664 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:38.198796 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:38.199517 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.200760 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.202824 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.203856 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.204081 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.207187 (+ 3106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:38.207428 (+ 241us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:38.207552 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:38.207862 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 16:44:38.207995 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:44:38.223915 734 raft_consensus.cc:1261] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Refusing update from remote peer e8b5a0fa71ae4b06b7358d092a3f20ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:38.224989 496 raft_consensus.cc:1261] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Refusing update from remote peer e8b5a0fa71ae4b06b7358d092a3f20ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:38.224913 1113 consensus_queue.cc:1035] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:38.226374 1113 consensus_queue.cc:1035] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:38.263680 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.258112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:58401 (local address 127.28.243.3:37837)
0812 16:44:38.258381 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:38.258388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:38.258406 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:44:38.258471 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:38.258476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:38.258559 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:38.258652 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:38.259255 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.260194 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.261740 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.262313 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.262432 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.262873 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:38.263035 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:38.263113 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:38.263253 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 16:44:38.263338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260812 16:44:38.263692 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:38.257438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:37837 (local address 127.28.243.2:58401)
0812 16:44:38.257831 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:38.257852 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:38.257972 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:38.258822 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:38.258827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:38.258841 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:38.259085 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.259100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.260375 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.260381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.261587 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:38.261601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:38.262476 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:38.262489 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:38.262663 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:38.262708 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:38.263127 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:38.263139 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 16:44:38.263222 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:44:38.263376 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
W20260812 16:44:38.266754 1147 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 16:44:39.901777 32655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:39.902050 32655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:39.909410 32655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Sending DeleteTablet for 3 replicas of tablet fe7c93f4acf1463b8428e44609a95165
I20260812 16:44:39.910902 947 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7c93f4acf1463b8428e44609a95165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:39 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260812 16:44:39.911116 694 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7c93f4acf1463b8428e44609a95165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:39 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:35666
I20260812 16:44:39.911108 449 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7c93f4acf1463b8428e44609a95165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:39 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60110
I20260812 16:44:39.912343 1263 tablet_replica.cc:330] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: stopping tablet replica
I20260812 16:44:39.912832 1263 raft_consensus.cc:2229] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:39.913255 1263 raft_consensus.cc:2258] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:39.914397 1264 tablet_replica.cc:330] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: stopping tablet replica
I20260812 16:44:39.915207 1261 tablet_replica.cc:330] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: stopping tablet replica
I20260812 16:44:39.915464 1263 ts_tablet_manager.cc:1905] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:39.915547 1261 raft_consensus.cc:2229] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:44:39.915803 1264 raft_consensus.cc:2229] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:39.916425 1261 raft_consensus.cc:2258] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:39.916687 1264 raft_consensus.cc:2258] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:39.917806 1264 ts_tablet_manager.cc:1905] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:39.919793 1261 ts_tablet_manager.cc:1905] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:39.923419 1263 ts_tablet_manager.cc:1918] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:44:39.923653 1263 log.cc:1196] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/fe7c93f4acf1463b8428e44609a95165
I20260812 16:44:39.924274 1263 ts_tablet_manager.cc:1939] T fe7c93f4acf1463b8428e44609a95165 P d355e2f44b824c258e73d8ea55fff0ff: Deleting consensus metadata
I20260812 16:44:39.925886 32630 catalog_manager.cc:4841] TS d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865): tablet fe7c93f4acf1463b8428e44609a95165 (table test-table [id=bcefea7eaa7242a69f9971bc44ad5f4b]) successfully deleted
I20260812 16:44:39.926075 1264 ts_tablet_manager.cc:1918] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:44:39.926234 1264 log.cc:1196] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/fe7c93f4acf1463b8428e44609a95165
I20260812 16:44:39.926679 1264 ts_tablet_manager.cc:1939] T fe7c93f4acf1463b8428e44609a95165 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting consensus metadata
I20260812 16:44:39.928064 32626 catalog_manager.cc:4841] TS c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209): tablet fe7c93f4acf1463b8428e44609a95165 (table test-table [id=bcefea7eaa7242a69f9971bc44ad5f4b]) successfully deleted
I20260812 16:44:39.928336 1261 ts_tablet_manager.cc:1918] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:44:39.928476 1261 log.cc:1196] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/fe7c93f4acf1463b8428e44609a95165
I20260812 16:44:39.929006 1261 ts_tablet_manager.cc:1939] T fe7c93f4acf1463b8428e44609a95165 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting consensus metadata
I20260812 16:44:39.930176 32630 catalog_manager.cc:4841] TS e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837): tablet fe7c93f4acf1463b8428e44609a95165 (table test-table [id=bcefea7eaa7242a69f9971bc44ad5f4b]) successfully deleted
Aug 12 16:44:39 dist-test-slave-1vmg krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:44:39 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:39.982009 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:39.963400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.28.243.62:45937)
0812 16:44:39.964355 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:39.964378 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:39.964413 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:44:39.964480 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:39.964484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:39.964556 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:39.964664 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:39.966519 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:39.967467 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:39.969555 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:39.970282 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:39.970473 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:39.976100 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:39.976108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:39.976112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:39.976159 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:39.979192 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:39.979991 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:39.979996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:39.979998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:39.980141 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:39.980509 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:39.980513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:39.980515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:39.980828 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:39.981078 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:39.981387 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 16:44:39.981642 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":524,"threads_started":1}
I20260812 16:44:39.987387 32655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:44:39.988281 32655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:39.998128 449 tablet_service.cc:1467] Processing CreateTablet for tablet d0f4f16c856f401aa52ac3ee0c30bad3 (DEFAULT_TABLE table=test-table [id=5d9c6d02fecc4fe2b474b518fbda30a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:39.998500 947 tablet_service.cc:1467] Processing CreateTablet for tablet d0f4f16c856f401aa52ac3ee0c30bad3 (DEFAULT_TABLE table=test-table [id=5d9c6d02fecc4fe2b474b518fbda30a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:39.998667 449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f4f16c856f401aa52ac3ee0c30bad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:39.998474 694 tablet_service.cc:1467] Processing CreateTablet for tablet d0f4f16c856f401aa52ac3ee0c30bad3 (DEFAULT_TABLE table=test-table [id=5d9c6d02fecc4fe2b474b518fbda30a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:39.999037 947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f4f16c856f401aa52ac3ee0c30bad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:39.999101 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f4f16c856f401aa52ac3ee0c30bad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:40.008103 1282 tablet_bootstrap.cc:492] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Bootstrap starting.
I20260812 16:44:40.009879 1283 tablet_bootstrap.cc:492] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Bootstrap starting.
I20260812 16:44:40.013785 1283 tablet_bootstrap.cc:654] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:40.014726 1281 tablet_bootstrap.cc:492] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Bootstrap starting.
I20260812 16:44:40.016764 1281 tablet_bootstrap.cc:654] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:40.016811 1282 tablet_bootstrap.cc:654] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:40.017189 1283 tablet_bootstrap.cc:492] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: No bootstrap required, opened a new log
I20260812 16:44:40.017418 1283 ts_tablet_manager.cc:1397] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260812 16:44:40.018347 1283 raft_consensus.cc:348] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.018707 1283 raft_consensus.cc:374] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:40.018806 1283 raft_consensus.cc:729] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Initialized, Role: FOLLOWER
I20260812 16:44:40.019105 1283 consensus_queue.cc:260] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.019793 1283 ts_tablet_manager.cc:1428] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:44:40.020054 1281 tablet_bootstrap.cc:492] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: No bootstrap required, opened a new log
I20260812 16:44:40.020232 1281 ts_tablet_manager.cc:1397] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 16:44:40.020504 1282 tablet_bootstrap.cc:492] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: No bootstrap required, opened a new log
I20260812 16:44:40.020669 1282 ts_tablet_manager.cc:1397] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20260812 16:44:40.021044 1281 raft_consensus.cc:348] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.021296 1281 raft_consensus.cc:374] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:40.021384 1281 raft_consensus.cc:729] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Initialized, Role: FOLLOWER
I20260812 16:44:40.021546 1282 raft_consensus.cc:348] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.021654 1281 consensus_queue.cc:260] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.021806 1282 raft_consensus.cc:374] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:40.021905 1282 raft_consensus.cc:729] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c880cc41f9804aa7bd285b0b16d91c2a, State: Initialized, Role: FOLLOWER
I20260812 16:44:40.022320 1281 ts_tablet_manager.cc:1428] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 16:44:40.022202 1282 consensus_queue.cc:260] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.024268 1282 ts_tablet_manager.cc:1428] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 16:44:40.043497 1220 raft_consensus.cc:482] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:40.043733 1220 raft_consensus.cc:504] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.044523 1220 leader_election.cc:288] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:40.045257 496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f4f16c856f401aa52ac3ee0c30bad3" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" is_pre_election: true
I20260812 16:44:40.045472 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f4f16c856f401aa52ac3ee0c30bad3" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d355e2f44b824c258e73d8ea55fff0ff" is_pre_election: true
I20260812 16:44:40.045969 734 raft_consensus.cc:2454] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 0.
I20260812 16:44:40.045817 496 raft_consensus.cc:2454] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 0.
I20260812 16:44:40.046787 865 leader_election.cc:302] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c880cc41f9804aa7bd285b0b16d91c2a, d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:40.047425 1220 raft_consensus.cc:2781] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:40.047556 1220 raft_consensus.cc:482] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:40.047632 1220 raft_consensus.cc:3037] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:40.050410 1220 raft_consensus.cc:504] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.051626 1220 leader_election.cc:288] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 election: Requested vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865)
I20260812 16:44:40.052691 496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f4f16c856f401aa52ac3ee0c30bad3" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
I20260812 16:44:40.052821 734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f4f16c856f401aa52ac3ee0c30bad3" candidate_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d355e2f44b824c258e73d8ea55fff0ff"
I20260812 16:44:40.053067 496 raft_consensus.cc:3037] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:40.053124 734 raft_consensus.cc:3037] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:40.055809 496 raft_consensus.cc:2454] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 1.
I20260812 16:44:40.055819 734 raft_consensus.cc:2454] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b5a0fa71ae4b06b7358d092a3f20ec in term 1.
I20260812 16:44:40.056880 865 leader_election.cc:302] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:40.057592 1220 raft_consensus.cc:2781] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:40.057986 1220 raft_consensus.cc:686] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 LEADER]: Becoming Leader. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Running, Role: LEADER
I20260812 16:44:40.058660 1220 consensus_queue.cc:237] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.062551 32658 catalog_manager.cc:5495] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec reported cstate change: term changed from 0 to 1, leader changed from <none> to e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3). New cstate: current_term: 1 leader_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } health_report { overall_health: HEALTHY } } }
I20260812 16:44:40.096448 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:40.088513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.28.243.3:37837)
0812 16:44:40.089618 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:40.089623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:40.089648 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:44:40.089717 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:40.089720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:40.089798 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:40.089906 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:40.090723 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.091761 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.094120 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.094672 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.094812 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:40.095627 (+ 815us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:40.095786 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:40.095865 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:40.096028 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 16:44:40.096111 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":780,"threads_started":1}
I20260812 16:44:40.107407 734 raft_consensus.cc:1261] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Refusing update from remote peer e8b5a0fa71ae4b06b7358d092a3f20ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:40.107770 496 raft_consensus.cc:1261] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Refusing update from remote peer e8b5a0fa71ae4b06b7358d092a3f20ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:40.108258 1255 consensus_queue.cc:1035] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 16:44:40.109071 1220 consensus_queue.cc:1035] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:40.181286 32658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:40.181542 32658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:40.186693 32658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Sending DeleteTablet for 3 replicas of tablet d0f4f16c856f401aa52ac3ee0c30bad3
I20260812 16:44:40.188100 694 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f4f16c856f401aa52ac3ee0c30bad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:40 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:35666
I20260812 16:44:40.188552 449 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f4f16c856f401aa52ac3ee0c30bad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:40 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60110
I20260812 16:44:40.188632 1263 tablet_replica.cc:330] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: stopping tablet replica
I20260812 16:44:40.188601 947 tablet_service.cc:1514] Processing DeleteTablet for tablet d0f4f16c856f401aa52ac3ee0c30bad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:40 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260812 16:44:40.188861 1264 tablet_replica.cc:330] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: stopping tablet replica
I20260812 16:44:40.188925 1261 tablet_replica.cc:330] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: stopping tablet replica
I20260812 16:44:40.188848 1263 raft_consensus.cc:2229] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:40.189182 1264 raft_consensus.cc:2229] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:40.189193 1263 raft_consensus.cc:2258] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:40.189128 1261 raft_consensus.cc:2229] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:44:40.189414 1264 raft_consensus.cc:2258] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:40.189821 1261 raft_consensus.cc:2258] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:40.191229 1264 ts_tablet_manager.cc:1905] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:40.191241 1263 ts_tablet_manager.cc:1905] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:40.191453 1261 ts_tablet_manager.cc:1905] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:40.192113 32658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:44:40.193063 32658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:44:40.200130 1264 ts_tablet_manager.cc:1918] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:40.200347 1264 log.cc:1196] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/d0f4f16c856f401aa52ac3ee0c30bad3
I20260812 16:44:40.200807 1264 ts_tablet_manager.cc:1939] T d0f4f16c856f401aa52ac3ee0c30bad3 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting consensus metadata
I20260812 16:44:40.200913 1261 ts_tablet_manager.cc:1918] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:40.201169 1261 log.cc:1196] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/d0f4f16c856f401aa52ac3ee0c30bad3
I20260812 16:44:40.201591 1261 ts_tablet_manager.cc:1939] T d0f4f16c856f401aa52ac3ee0c30bad3 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting consensus metadata
I20260812 16:44:40.202744 32626 catalog_manager.cc:4841] TS c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209): tablet d0f4f16c856f401aa52ac3ee0c30bad3 (table test-table [id=5d9c6d02fecc4fe2b474b518fbda30a4]) successfully deleted
I20260812 16:44:40.202950 947 tablet_service.cc:1467] Processing CreateTablet for tablet c73ebd15f8024285ade39bcb61abfac4 (DEFAULT_TABLE table=test-table [id=a01d0168869943d29343d25238228dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:40.202850 694 tablet_service.cc:1467] Processing CreateTablet for tablet c73ebd15f8024285ade39bcb61abfac4 (DEFAULT_TABLE table=test-table [id=a01d0168869943d29343d25238228dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:40.203423 947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73ebd15f8024285ade39bcb61abfac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:40.203377 449 tablet_service.cc:1467] Processing CreateTablet for tablet c73ebd15f8024285ade39bcb61abfac4 (DEFAULT_TABLE table=test-table [id=a01d0168869943d29343d25238228dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:44:40.203593 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73ebd15f8024285ade39bcb61abfac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:40.203826 449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73ebd15f8024285ade39bcb61abfac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:40.204346 1263 ts_tablet_manager.cc:1918] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:44:40.204684 1263 log.cc:1196] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/d0f4f16c856f401aa52ac3ee0c30bad3
I20260812 16:44:40.205291 1263 ts_tablet_manager.cc:1939] T d0f4f16c856f401aa52ac3ee0c30bad3 P d355e2f44b824c258e73d8ea55fff0ff: Deleting consensus metadata
I20260812 16:44:40.207131 32630 catalog_manager.cc:4841] TS d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2:36865): tablet d0f4f16c856f401aa52ac3ee0c30bad3 (table test-table [id=5d9c6d02fecc4fe2b474b518fbda30a4]) successfully deleted
I20260812 16:44:40.207422 32630 catalog_manager.cc:4841] TS e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837): tablet d0f4f16c856f401aa52ac3ee0c30bad3 (table test-table [id=5d9c6d02fecc4fe2b474b518fbda30a4]) successfully deleted
I20260812 16:44:40.213675 1283 tablet_bootstrap.cc:492] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Bootstrap starting.
I20260812 16:44:40.213850 1281 tablet_bootstrap.cc:492] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: Bootstrap starting.
I20260812 16:44:40.215015 1282 tablet_bootstrap.cc:492] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: Bootstrap starting.
I20260812 16:44:40.215813 1283 tablet_bootstrap.cc:654] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:40.216051 1281 tablet_bootstrap.cc:654] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:40.216886 1282 tablet_bootstrap.cc:654] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:40.219543 1281 tablet_bootstrap.cc:492] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: No bootstrap required, opened a new log
I20260812 16:44:40.219692 1282 tablet_bootstrap.cc:492] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: No bootstrap required, opened a new log
I20260812 16:44:40.219791 1281 ts_tablet_manager.cc:1397] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 16:44:40.219892 1282 ts_tablet_manager.cc:1397] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 16:44:40.220647 1283 tablet_bootstrap.cc:492] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: No bootstrap required, opened a new log
I20260812 16:44:40.220633 1281 raft_consensus.cc:348] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.220932 1283 ts_tablet_manager.cc:1397] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260812 16:44:40.220954 1281 raft_consensus.cc:374] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:40.221042 1281 raft_consensus.cc:729] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Initialized, Role: FOLLOWER
I20260812 16:44:40.220897 1282 raft_consensus.cc:348] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.221160 1282 raft_consensus.cc:374] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:40.221237 1282 raft_consensus.cc:729] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c880cc41f9804aa7bd285b0b16d91c2a, State: Initialized, Role: FOLLOWER
I20260812 16:44:40.221289 1281 consensus_queue.cc:260] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.221606 1282 consensus_queue.cc:260] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.221834 1281 ts_tablet_manager.cc:1428] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 16:44:40.221822 1283 raft_consensus.cc:348] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.222054 1283 raft_consensus.cc:374] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:40.222146 1282 ts_tablet_manager.cc:1428] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 16:44:40.222144 1283 raft_consensus.cc:729] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b5a0fa71ae4b06b7358d092a3f20ec, State: Initialized, Role: FOLLOWER
I20260812 16:44:40.222390 1283 consensus_queue.cc:260] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.223009 1283 ts_tablet_manager.cc:1428] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:44:40.302418 1222 raft_consensus.cc:482] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:44:40.302640 1222 raft_consensus.cc:504] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.303478 1222 leader_election.cc:288] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837)
I20260812 16:44:40.304368 984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73ebd15f8024285ade39bcb61abfac4" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" is_pre_election: true
I20260812 16:44:40.304412 494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73ebd15f8024285ade39bcb61abfac4" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" is_pre_election: true
I20260812 16:44:40.304636 984 raft_consensus.cc:2454] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 0.
I20260812 16:44:40.304860 494 raft_consensus.cc:2454] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 0.
I20260812 16:44:40.305285 609 leader_election.cc:302] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:40.305617 1222 raft_consensus.cc:2781] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:44:40.305752 1222 raft_consensus.cc:482] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:44:40.305841 1222 raft_consensus.cc:3037] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:40.308619 1222 raft_consensus.cc:504] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.309533 1222 leader_election.cc:288] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 election: Requested vote from peers c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209), e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837)
I20260812 16:44:40.310499 984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73ebd15f8024285ade39bcb61abfac4" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec"
I20260812 16:44:40.310791 984 raft_consensus.cc:3037] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:40.310670 494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73ebd15f8024285ade39bcb61abfac4" candidate_uuid: "d355e2f44b824c258e73d8ea55fff0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c880cc41f9804aa7bd285b0b16d91c2a"
I20260812 16:44:40.311012 494 raft_consensus.cc:3037] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:40.313802 984 raft_consensus.cc:2454] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 1.
I20260812 16:44:40.313796 494 raft_consensus.cc:2454] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d355e2f44b824c258e73d8ea55fff0ff in term 1.
I20260812 16:44:40.314587 603 leader_election.cc:302] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c880cc41f9804aa7bd285b0b16d91c2a, d355e2f44b824c258e73d8ea55fff0ff, e8b5a0fa71ae4b06b7358d092a3f20ec; no voters:
I20260812 16:44:40.315166 1222 raft_consensus.cc:2781] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:40.315416 1222 raft_consensus.cc:686] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 1 LEADER]: Becoming Leader. State: Replica: d355e2f44b824c258e73d8ea55fff0ff, State: Running, Role: LEADER
I20260812 16:44:40.315744 1222 consensus_queue.cc:237] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } }
I20260812 16:44:40.319315 32651 catalog_manager.cc:5495] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff reported cstate change: term changed from 0 to 1, leader changed from <none> to d355e2f44b824c258e73d8ea55fff0ff (127.28.243.2). New cstate: current_term: 1 leader_uuid: "d355e2f44b824c258e73d8ea55fff0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d355e2f44b824c258e73d8ea55fff0ff" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 36865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20260812 16:44:40.369073 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:40.361760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.28.243.2:36865)
0812 16:44:40.362572 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:40.362577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:40.362599 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:44:40.362659 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:40.362663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:40.362731 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:40.362835 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:40.363329 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.364467 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.366449 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.367174 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.367341 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:40.367993 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:40.368154 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:40.368244 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:40.368451 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 16:44:40.368563 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":412,"threads_started":1}
I20260812 16:44:40.386858 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:40.380628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:41819 (local address 127.28.243.2:36865)
0812 16:44:40.380807 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:40.380815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:40.381155 (+ 340us) server_negotiation.cc:407] Connection header received
0812 16:44:40.381334 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:40.381338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:40.381402 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:40.381481 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:40.381961 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.382736 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.384585 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.385124 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.385309 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:40.385900 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:40.386061 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:40.386142 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:40.386448 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 16:44:40.386547 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:44:40.387064 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:40.380394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.28.243.2:41819)
0812 16:44:40.381029 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:40.381046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:40.381176 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:40.381564 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:40.381568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:40.381578 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:44:40.381797 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:40.381805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.382881 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.382886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:40.384207 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:40.384221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:40.385470 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:40.385477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:40.385639 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:40.385696 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:44:40.386232 (+ 536us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:44:40.386252 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 16:44:40.386356 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 16:44:40.386511 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":333,"threads_started":1}
I20260812 16:44:40.397090 494 raft_consensus.cc:1261] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Refusing update from remote peer d355e2f44b824c258e73d8ea55fff0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:40.397091 984 raft_consensus.cc:1261] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Refusing update from remote peer d355e2f44b824c258e73d8ea55fff0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:40.397728 1253 consensus_queue.cc:1035] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b5a0fa71ae4b06b7358d092a3f20ec" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:40.398111 1222 consensus_queue.cc:1035] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c880cc41f9804aa7bd285b0b16d91c2a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:44:40.424054 1223 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 16:44:42.068483 32651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:42.068738 32651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:44:42.074604 32651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1b53136acee45cb884d6ae850f2f601: Sending DeleteTablet for 3 replicas of tablet c73ebd15f8024285ade39bcb61abfac4
I20260812 16:44:42.075773 449 tablet_service.cc:1514] Processing DeleteTablet for tablet c73ebd15f8024285ade39bcb61abfac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:42 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60110
I20260812 16:44:42.075791 694 tablet_service.cc:1514] Processing DeleteTablet for tablet c73ebd15f8024285ade39bcb61abfac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:42 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:35666
I20260812 16:44:42.075886 947 tablet_service.cc:1514] Processing DeleteTablet for tablet c73ebd15f8024285ade39bcb61abfac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:44:42 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260812 16:44:42.076373 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 325
I20260812 16:44:42.092031 1365 tablet_replica.cc:330] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: stopping tablet replica
I20260812 16:44:42.092581 1365 raft_consensus.cc:2229] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:44:42.093848 1365 raft_consensus.cc:2258] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:42.094455 1366 tablet_replica.cc:330] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: stopping tablet replica
I20260812 16:44:42.094723 1366 raft_consensus.cc:2229] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:42.095201 1366 raft_consensus.cc:2258] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:42.096081 1365 ts_tablet_manager.cc:1905] T c73ebd15f8024285ade39bcb61abfac4 P d355e2f44b824c258e73d8ea55fff0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:42.096107 1364 tablet_replica.cc:330] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: stopping tablet replica
I20260812 16:44:42.096362 1364 raft_consensus.cc:2229] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:44:42.096664 1364 raft_consensus.cc:2258] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:44:42.096801 1366 ts_tablet_manager.cc:1905] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:44:42.097931 1364 ts_tablet_manager.cc:1905] T c73ebd15f8024285ade39bcb61abfac4 P c880cc41f9804aa7bd285b0b16d91c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 16:44:42 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:42.119025 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.077874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.28.243.1:35209)
0812 16:44:42.089607 (+ 11733us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:42.089616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:42.089659 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:44:42.089763 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:42.089769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:42.089884 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:42.090050 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:42.093982 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:42.096124 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:42.098826 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:42.099574 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:42.099727 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:42.110599 (+ 10872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:42.110672 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:42.110696 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:42.110857 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:42.115323 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:42.116191 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:42.116202 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:42.116206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:42.116405 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:42.116939 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:42.116945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:42.116948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:42.117612 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:42.117943 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:42.118283 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 16:44:42.118549 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11503,"thread_start_us":11313,"threads_started":1}
W20260812 16:44:42.120008 559 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60126
W20260812 16:44:42.120709 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:42.120995 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 325
I20260812 16:44:42.142429 1366 ts_tablet_manager.cc:1918] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:44:42.142617 1366 log.cc:1196] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/c73ebd15f8024285ade39bcb61abfac4
I20260812 16:44:42.143165 1366 ts_tablet_manager.cc:1939] T c73ebd15f8024285ade39bcb61abfac4 P e8b5a0fa71ae4b06b7358d092a3f20ec: Deleting consensus metadata
W20260812 16:44:42.178833 32626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:35209 (error 108)
I20260812 16:44:42.179281 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 578
I20260812 16:44:42.182175 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.180613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60136)
0812 16:44:42.181464 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.181828 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":374,"threads_started":1}
W20260812 16:44:42.182461 32626 catalog_manager.cc:4584] TS c880cc41f9804aa7bd285b0b16d91c2a (127.28.243.1:35209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c73ebd15f8024285ade39bcb61abfac4: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Aug 12 16:44:42 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:44:42.201841 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.180171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.28.243.2:36865)
0812 16:44:42.180744 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:42.180749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:42.180771 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:44:42.180841 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:42.180847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:42.180942 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:42.181073 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:42.183764 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:42.185480 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:42.187120 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:42.187700 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:42.188156 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:42.194697 (+ 6541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:42.194718 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:42.194737 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:42.194785 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:42.198267 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:42.199356 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:42.199369 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:42.199372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:42.199658 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:42.200238 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:42.200245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:42.200247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:42.200709 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:42.200967 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:42.201336 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 16:44:42.201531 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":582,"threads_started":1}
W20260812 16:44:42.202592 820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35690
W20260812 16:44:42.203512 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:42.203688 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 578
I20260812 16:44:42.227326 32630 catalog_manager.cc:4841] TS e8b5a0fa71ae4b06b7358d092a3f20ec (127.28.243.3:37837): tablet c73ebd15f8024285ade39bcb61abfac4 (table test-table [id=a01d0168869943d29343d25238228dd0]) successfully deleted
I20260812 16:44:42.236929 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.236347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60150)
0812 16:44:42.236602 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.236739 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 16:44:42.238267 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.237808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60162)
0812 16:44:42.237966 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.238081 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 16:44:42.267122 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 836
I20260812 16:44:42.267700 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.267195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.0.0.1:35692)
0812 16:44:42.267428 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.267507 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.2:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
Aug 12 16:44:42 dist-test-slave-1vmg krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:44:42.285772 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.268149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.28.243.3:37837)
0812 16:44:42.269777 (+ 1628us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:42.269783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:42.269809 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:44:42.269889 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:42.269895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:42.270021 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:42.270153 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:42.271857 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:42.272705 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:42.274156 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:42.274610 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:42.274721 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:42.279380 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:42.279402 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:42.279420 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:42.279463 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:42.282593 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:42.283481 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:42.283489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:42.283491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:42.283647 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:42.284126 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:42.284130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:42.284132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:42.284573 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:42.284836 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:42.285276 (+ 440us) server_negotiation.cc:299] Negotiation successful
0812 16:44:42.285478 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":691,"threads_started":1}
W20260812 16:44:42.286783 1060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46442
W20260812 16:44:42.287809 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 836: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:42.288007 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 836
I20260812 16:44:42.299932 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.299304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60170)
0812 16:44:42.299573 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.299686 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 16:44:42.300426 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.299928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.0.0.1:35698)
0812 16:44:42.300199 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.300283 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.2:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"threads_started":1}
I20260812 16:44:42.301671 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.301238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.0.0.1:35702)
0812 16:44:42.301448 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.301515 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.2:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":32}
I20260812 16:44:42.301892 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.301227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60178)
0812 16:44:42.301445 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.301663 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 16:44:42.327155 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32603
I20260812 16:44:42.328827 32741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:42.468859 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.468379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35209 (local address 127.0.0.1:60184)
0812 16:44:42.468585 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.468699 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:44:42.469257 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:42.468516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:36865 (local address 127.0.0.1:35708)
0812 16:44:42.468876 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:42.468977 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.2:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260812 16:44:42.469568 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 32603
2026-08-12T16:44:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9754 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:42 dist-test-slave-1vmg krb5kdc[1386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:42 dist-test-slave-1vmg krb5kdc[1386](info): set up 2 sockets
Aug 12 16:44:42 dist-test-slave-1vmg krb5kdc[1386](info): commencing operation
krb5kdc: starting...
W20260812 16:44:43.750437 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:43 dist-test-slave-1vmg krb5kdc[1386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:44.008332 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:37295
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:44.380357 1405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:44.380678 1405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:44.380797 1405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:44.389063 1405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:44.389214 1405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:44.389259 1405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:44.389310 1405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:44.389346 1405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:44.400161 1405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:37295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:44.401647 1405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:44.403126 1405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:44.417351 1420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:44.417389 1423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:44.417405 1417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:44.418349 1405 server_base.cc:1034] running on GCE node
I20260812 16:44:44.419250 1405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:44.420819 1405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:44.422086 1405 hybrid_clock.cc:648] HybridClock initialized: now 1786553084422022 us; error 52 us; skew 500 ppm
I20260812 16:44:44.422451 1405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:44 dist-test-slave-1vmg krb5kdc[1386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553084, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:44.426451 1405 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:44.428104 1405 webserver.cc:460] Webserver started at http://127.28.243.62:42043/ using document root <none> and password file <none>
I20260812 16:44:44.428864 1405 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:44.428987 1405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:44.429316 1405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:44.432693 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "d6cff5ec467242b887f5551ba81b4b5d"
format_stamp: "Formatted at 2026-08-12 16:44:44 on dist-test-slave-1vmg"
server_key: "4c14a19a33bfff5db834fc1841ce42d0"
server_key_iv: "bc47c0365faaf81d26fdeda417720f28"
server_key_version: "encryptionkey@0"
I20260812 16:44:44.433537 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "d6cff5ec467242b887f5551ba81b4b5d"
format_stamp: "Formatted at 2026-08-12 16:44:44 on dist-test-slave-1vmg"
server_key: "4c14a19a33bfff5db834fc1841ce42d0"
server_key_iv: "bc47c0365faaf81d26fdeda417720f28"
server_key_version: "encryptionkey@0"
I20260812 16:44:44.439956 1405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260812 16:44:44.444591 1434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:44.446682 1405 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:44:44.446897 1405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "d6cff5ec467242b887f5551ba81b4b5d"
format_stamp: "Formatted at 2026-08-12 16:44:44 on dist-test-slave-1vmg"
server_key: "4c14a19a33bfff5db834fc1841ce42d0"
server_key_iv: "bc47c0365faaf81d26fdeda417720f28"
server_key_version: "encryptionkey@0"
I20260812 16:44:44.447177 1405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:44.477249 1405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:44.481933 1405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:44.482237 1405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:44.503865 1405 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:37295
I20260812 16:44:44.503887 1534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:37295 every 8 connection(s)
I20260812 16:44:44.505368 1405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:44.506552 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1405
I20260812 16:44:44.506793 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:44.507448 29644 external_mini_cluster.cc:1428] Setting key 663e8bb01995d577921ed6326be468fa
I20260812 16:44:44.512253 1535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 16:44:44 dist-test-slave-1vmg krb5kdc[1386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:44.529793 1535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d: Bootstrap starting.
I20260812 16:44:44.534313 1535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:44.534991 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:44.509747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.28.243.62:37295)
0812 16:44:44.510606 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:44.510620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:44.510671 (+ 51us) server_negotiation.cc:407] Connection header received
0812 16:44:44.512529 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:44.512557 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:44.513173 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:44.513630 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:44.516185 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:44.517661 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:44.519711 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:44.520482 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:44.520660 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:44.526712 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:44.526722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:44.526743 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:44.526793 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:44.530521 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:44.531360 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:44.531367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:44.531374 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:44.531666 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:44.532049 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:44.532053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:44.532055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:44.532655 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:44.532971 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:44.533400 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 16:44:44.533853 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":362,"threads_started":1}
I20260812 16:44:44.535717 1535 log.cc:824] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:44.538894 1535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d: No bootstrap required, opened a new log
I20260812 16:44:44.548651 1535 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6cff5ec467242b887f5551ba81b4b5d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37295 } }
I20260812 16:44:44.549111 1535 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:44.549190 1535 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cff5ec467242b887f5551ba81b4b5d, State: Initialized, Role: FOLLOWER
I20260812 16:44:44.549994 1535 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6cff5ec467242b887f5551ba81b4b5d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37295 } }
I20260812 16:44:44.550374 1535 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:44.550657 1535 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:44.550874 1535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:44.553480 1535 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6cff5ec467242b887f5551ba81b4b5d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37295 } }
I20260812 16:44:44.554055 1535 leader_election.cc:302] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6cff5ec467242b887f5551ba81b4b5d; no voters:
I20260812 16:44:44.554695 1535 leader_election.cc:288] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:44.554860 1542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:44.555840 1542 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [term 1 LEADER]: Becoming Leader. State: Replica: d6cff5ec467242b887f5551ba81b4b5d, State: Running, Role: LEADER
I20260812 16:44:44.556408 1542 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6cff5ec467242b887f5551ba81b4b5d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37295 } }
I20260812 16:44:44.557372 1535 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:44.559387 1543 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6cff5ec467242b887f5551ba81b4b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6cff5ec467242b887f5551ba81b4b5d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37295 } } }
I20260812 16:44:44.559828 1543 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:44.560559 1544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6cff5ec467242b887f5551ba81b4b5d. Latest consensus state: current_term: 1 leader_uuid: "d6cff5ec467242b887f5551ba81b4b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6cff5ec467242b887f5551ba81b4b5d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37295 } } }
I20260812 16:44:44.560856 1544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:44.560956 1549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:44.567688 1549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:44.579260 1549 catalog_manager.cc:1347] Generated new cluster ID: c10719d5429c41c4bfb230c1e08f6530
I20260812 16:44:44.579406 1549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:44.601769 1549 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:44.603240 1549 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:44.620282 1549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6cff5ec467242b887f5551ba81b4b5d: Generated new TSK 0
I20260812 16:44:44.621474 1549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:44.682909 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:37295
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:45.048640 1571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:45.048969 1571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:45.049067 1571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:45.057153 1571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:45.057297 1571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:45.057457 1571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:45.068374 1571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:37295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:45.069734 1571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:45.071110 1571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:45.084853 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:45.084973 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:45.085002 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:45.086127 1571 server_base.cc:1034] running on GCE node
I20260812 16:44:45.086741 1571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:45.087738 1571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:45.088938 1571 hybrid_clock.cc:648] HybridClock initialized: now 1786553085088878 us; error 48 us; skew 500 ppm
I20260812 16:44:45.089224 1571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:45 dist-test-slave-1vmg krb5kdc[1386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:45.092326 1571 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:45.093878 1571 webserver.cc:460] Webserver started at http://127.28.243.1:41591/ using document root <none> and password file <none>
I20260812 16:44:45.094601 1571 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:45.094707 1571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:45.095068 1571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:45.098038 1571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "0c7f26d3ce1143cea5b9208390ff8df7"
format_stamp: "Formatted at 2026-08-12 16:44:45 on dist-test-slave-1vmg"
server_key: "8a6cbb44fb223a91fd94f87594ca0911"
server_key_iv: "67b9b0865b0d6d66560392a182bdc4eb"
server_key_version: "encryptionkey@0"
I20260812 16:44:45.098809 1571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "0c7f26d3ce1143cea5b9208390ff8df7"
format_stamp: "Formatted at 2026-08-12 16:44:45 on dist-test-slave-1vmg"
server_key: "8a6cbb44fb223a91fd94f87594ca0911"
server_key_iv: "67b9b0865b0d6d66560392a182bdc4eb"
server_key_version: "encryptionkey@0"
I20260812 16:44:45.104766 1571 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 16:44:45.108681 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:45.110298 1571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:44:45.110462 1571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "0c7f26d3ce1143cea5b9208390ff8df7"
format_stamp: "Formatted at 2026-08-12 16:44:45 on dist-test-slave-1vmg"
server_key: "8a6cbb44fb223a91fd94f87594ca0911"
server_key_iv: "67b9b0865b0d6d66560392a182bdc4eb"
server_key_version: "encryptionkey@0"
I20260812 16:44:45.110658 1571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:45.123596 1571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:45.128049 1571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:45.128314 1571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:45.129295 1571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:45.130682 1571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:45.130756 1571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:45.130848 1571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:45.130892 1571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:45.166347 1571 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:32943
I20260812 16:44:45.166388 1798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:32943 every 8 connection(s)
I20260812 16:44:45.167670 1571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:45.168249 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1571
I20260812 16:44:45.168489 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:45.169504 29644 external_mini_cluster.cc:1428] Setting key a046916ed10810bbd7bed25fbee0233b
Aug 12 16:44:45 dist-test-slave-1vmg krb5kdc[1386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:45.190881 1801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:45.171137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44309 (local address 127.28.243.62:37295)
0812 16:44:45.171637 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:45.171642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:45.172990 (+ 1348us) server_negotiation.cc:407] Connection header received
0812 16:44:45.175007 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:45.175013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:45.175098 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:45.175201 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:45.176828 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.177778 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.179493 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.180043 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.180167 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:45.184614 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:45.184619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:45.184622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:45.184656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:45.187193 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:45.188298 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:45.188303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:45.188305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:45.188436 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:45.188922 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:45.188925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:45.188927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:45.189246 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:45.189367 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:45.190429 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0812 16:44:45.190624 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
I20260812 16:44:45.192996 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:45.171885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:37295 (local address 127.28.243.1:44309)
0812 16:44:45.172796 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:45.172849 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:45.174709 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:45.175438 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:45.175448 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:45.176041 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:45.176647 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:45.176666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.177958 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.177963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:45.179329 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:45.179344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.180303 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.180312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:45.180633 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:45.181406 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:45.181441 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:45.184362 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:45.187475 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:45.187487 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:45.187510 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:45.188136 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:45.188543 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:45.188548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:45.188552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:45.188813 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:45.189460 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:45.189470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:45.189961 (+ 491us) client_negotiation.cc:769] Sending connection context
0812 16:44:45.190293 (+ 332us) client_negotiation.cc:240] Negotiation successful
0812 16:44:45.191344 (+ 1051us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":349,"threads_started":1}
I20260812 16:44:45.194754 1799 heartbeater.cc:347] Connected to a master server at 127.28.243.62:37295
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:44:45.195088 1799 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:45.195837 1799 heartbeater.cc:511] Master 127.28.243.62:37295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
I20260812 16:44:45.198051 1467 ts_manager.cc:194] Registered new tserver with Master: 0c7f26d3ce1143cea5b9208390ff8df7 (127.28.243.1:32943)
I20260812 16:44:45.199573 1467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:44309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:45.236544 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:37295
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:45.630261 1811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:45.630586 1811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:45.630710 1811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:45.638520 1811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:45.638671 1811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:45.638863 1811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:45.649972 1811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:37295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:45.651571 1811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:45.653045 1811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:45.667224 1820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:45.667276 1824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:45.667330 1819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:45.668226 1811 server_base.cc:1034] running on GCE node
I20260812 16:44:45.669224 1811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:45.670598 1811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:45.671847 1811 hybrid_clock.cc:648] HybridClock initialized: now 1786553085671790 us; error 44 us; skew 500 ppm
I20260812 16:44:45.672348 1811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:45 dist-test-slave-1vmg krb5kdc[1386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:45.675781 1811 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:45.677379 1811 webserver.cc:460] Webserver started at http://127.28.243.2:39161/ using document root <none> and password file <none>
I20260812 16:44:45.678097 1811 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:45.678206 1811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:45.678489 1811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:45.681641 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "3e6013ec872f4483a77914c7e446e29a"
format_stamp: "Formatted at 2026-08-12 16:44:45 on dist-test-slave-1vmg"
server_key: "922d20c9864bb62e42c0bc2a5fafd563"
server_key_iv: "ccae61457de52a53414eb3aacb2b0f68"
server_key_version: "encryptionkey@0"
I20260812 16:44:45.682467 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "3e6013ec872f4483a77914c7e446e29a"
format_stamp: "Formatted at 2026-08-12 16:44:45 on dist-test-slave-1vmg"
server_key: "922d20c9864bb62e42c0bc2a5fafd563"
server_key_iv: "ccae61457de52a53414eb3aacb2b0f68"
server_key_version: "encryptionkey@0"
I20260812 16:44:45.690850 1811 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260812 16:44:45.694890 1838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:45.696564 1811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:44:45.696779 1811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "3e6013ec872f4483a77914c7e446e29a"
format_stamp: "Formatted at 2026-08-12 16:44:45 on dist-test-slave-1vmg"
server_key: "922d20c9864bb62e42c0bc2a5fafd563"
server_key_iv: "ccae61457de52a53414eb3aacb2b0f68"
server_key_version: "encryptionkey@0"
I20260812 16:44:45.697000 1811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:45.722507 1811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:45.729496 1811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:45.729899 1811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:45.731680 1811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:45.733457 1811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:45.733565 1811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:45.733697 1811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:45.733769 1811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:45.773301 1811 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:35975
I20260812 16:44:45.773341 2036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:35975 every 8 connection(s)
I20260812 16:44:45.774788 1811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:45.782083 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1811
I20260812 16:44:45.782308 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:45.782760 29644 external_mini_cluster.cc:1428] Setting key b8070ae3ac619c0468ea96007585ff49
Aug 12 16:44:45 dist-test-slave-1vmg krb5kdc[1386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:45.803601 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:45.778186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:56993 (local address 127.28.243.62:37295)
0812 16:44:45.778684 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:45.778691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:45.780102 (+ 1411us) server_negotiation.cc:407] Connection header received
0812 16:44:45.782790 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:45.782797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:45.782906 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:45.783179 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:45.785268 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.786025 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.788690 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.789291 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.789410 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:45.795893 (+ 6483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:45.795901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:45.795905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:45.795951 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:45.799016 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:45.800200 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:45.800209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:45.800212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:45.800516 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:45.801156 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:45.801162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:45.801164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:45.801524 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:45.801675 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:45.802749 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0812 16:44:45.803170 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260812 16:44:45.805506 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:45.778670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:37295 (local address 127.28.243.2:56993)
0812 16:44:45.779834 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:45.779908 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:45.782337 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:45.783450 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:45.783461 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:45.784313 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:45.785017 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:45.785047 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.786293 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.786298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:45.788457 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:45.788480 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:45.789606 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:45.789757 (+ 151us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:45.790259 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:45.791638 (+ 1379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:45.791685 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:45.795502 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:45.799213 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:45.799225 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:45.799245 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:45.799908 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:45.800669 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:45.800674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:45.800678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:45.800989 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:45.801690 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:45.801699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:45.802278 (+ 579us) client_negotiation.cc:769] Sending connection context
0812 16:44:45.802600 (+ 322us) client_negotiation.cc:240] Negotiation successful
0812 16:44:45.803231 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":439,"threads_started":1}
I20260812 16:44:45.807356 2037 heartbeater.cc:347] Connected to a master server at 127.28.243.62:37295
I20260812 16:44:45.807799 2037 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:45.808820 2037 heartbeater.cc:511] Master 127.28.243.62:37295 requested a full tablet report, sending...
I20260812 16:44:45.812346 1467 ts_manager.cc:194] Registered new tserver with Master: 3e6013ec872f4483a77914c7e446e29a (127.28.243.2:35975)
I20260812 16:44:45.813223 1467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:56993
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:45.851750 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:37295
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:46.203259 1799 heartbeater.cc:503] Master 127.28.243.62:37295 was elected leader, sending a full tablet report...
W20260812 16:44:46.248446 2043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:46.248797 2043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:46.248940 2043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:46.257925 2043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:46.258072 2043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:46.258232 2043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:46.270570 2043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:37295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:46.272768 2043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:46.274513 2043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:46.289709 2051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:46.289644 2057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:46.289692 2054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:46.290840 2043 server_base.cc:1034] running on GCE node
I20260812 16:44:46.291513 2043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:46.292567 2043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:46.293828 2043 hybrid_clock.cc:648] HybridClock initialized: now 1786553086293754 us; error 63 us; skew 500 ppm
I20260812 16:44:46.294202 2043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:46 dist-test-slave-1vmg krb5kdc[1386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:46.297695 2043 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:46.299399 2043 webserver.cc:460] Webserver started at http://127.28.243.3:34159/ using document root <none> and password file <none>
I20260812 16:44:46.300173 2043 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:46.300285 2043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:46.300567 2043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:46.304128 2043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "3ab7d0a83daa43f5a4a69211cd0c02cd"
format_stamp: "Formatted at 2026-08-12 16:44:46 on dist-test-slave-1vmg"
server_key: "20aea409907d49177e5291da67be09f0"
server_key_iv: "4e030e8bf781d17eed315d0f00e74db2"
server_key_version: "encryptionkey@0"
I20260812 16:44:46.305030 2043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "3ab7d0a83daa43f5a4a69211cd0c02cd"
format_stamp: "Formatted at 2026-08-12 16:44:46 on dist-test-slave-1vmg"
server_key: "20aea409907d49177e5291da67be09f0"
server_key_iv: "4e030e8bf781d17eed315d0f00e74db2"
server_key_version: "encryptionkey@0"
I20260812 16:44:46.310997 2043 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:44:46.315420 2068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:46.317301 2043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 16:44:46.317477 2043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "3ab7d0a83daa43f5a4a69211cd0c02cd"
format_stamp: "Formatted at 2026-08-12 16:44:46 on dist-test-slave-1vmg"
server_key: "20aea409907d49177e5291da67be09f0"
server_key_iv: "4e030e8bf781d17eed315d0f00e74db2"
server_key_version: "encryptionkey@0"
I20260812 16:44:46.317679 2043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:46.347296 2043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:46.352118 2043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:46.352382 2043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:46.353345 2043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:46.354658 2043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:46.354732 2043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:46.354822 2043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:46.354864 2043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:46.389760 2043 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:39551
I20260812 16:44:46.389808 2269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:39551 every 8 connection(s)
I20260812 16:44:46.391265 2043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:46.400632 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2043
I20260812 16:44:46.400844 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:46.401386 29644 external_mini_cluster.cc:1428] Setting key 0a848e23ba57633d5478bbf04d9423da
Aug 12 16:44:46 dist-test-slave-1vmg krb5kdc[1386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:46.417584 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:46.394872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:33455 (local address 127.28.243.62:37295)
0812 16:44:46.395460 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:46.395464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:46.396608 (+ 1144us) server_negotiation.cc:407] Connection header received
0812 16:44:46.398471 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:46.398475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:46.398549 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:46.398659 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:46.400360 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:46.401530 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:46.403811 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:46.404342 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:46.404448 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:46.409546 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:46.409552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:46.409556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:46.409595 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:46.412830 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:46.414443 (+ 1613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:46.414449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:46.414452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:46.414678 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:46.415397 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:46.415402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:46.415404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:46.415759 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:46.415940 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:46.416887 (+ 947us) server_negotiation.cc:299] Negotiation successful
0812 16:44:46.417168 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":273,"threads_started":1}
I20260812 16:44:46.418737 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:46.395568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:37295 (local address 127.28.243.3:33455)
0812 16:44:46.396405 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:46.396457 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:46.398163 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:46.398867 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:46.398877 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:46.399495 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:46.400156 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:46.400184 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:46.401725 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:46.401730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:46.403445 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:46.403464 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:46.404702 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:46.404714 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:46.405048 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:46.405889 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:46.405915 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:46.409238 (+ 3323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:46.413194 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:46.413207 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:46.413241 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:46.414072 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:46.414905 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:46.414927 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:46.414932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:46.415251 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:46.416011 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:46.416019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:46.416424 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 16:44:46.416712 (+ 288us) client_negotiation.cc:240] Negotiation successful
0812 16:44:46.417344 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":331,"threads_started":1}
I20260812 16:44:46.420400 2271 heartbeater.cc:347] Connected to a master server at 127.28.243.62:37295
I20260812 16:44:46.420746 2271 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:46.421579 2271 heartbeater.cc:511] Master 127.28.243.62:37295 requested a full tablet report, sending...
I20260812 16:44:46.423610 1467 ts_manager.cc:194] Registered new tserver with Master: 3ab7d0a83daa43f5a4a69211cd0c02cd (127.28.243.3:39551)
I20260812 16:44:46.424474 1467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:33455
I20260812 16:44:46.432550 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 16:44:46.475447 2274 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:46.472726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.28.243.62:37295)
0812 16:44:46.472950 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:46.472958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:46.473312 (+ 354us) server_negotiation.cc:407] Connection header received
0812 16:44:46.473481 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:46.473485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:46.473573 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:46.473661 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:46.475160 (+ 1499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56444: BlockingRecv error: recv got EOF from 127.0.0.1:56444 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:44:46.478523 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1571
W20260812 16:44:46.481741 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1571: Not authorized: Client connection negotiation failed: client connection to 127.28.243.1:32943: server requires authentication, but client does not have Kerberos credentials available
I20260812 16:44:46.481907 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1571
I20260812 16:44:46.505529 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1811
W20260812 16:44:46.508793 2297 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:46.506468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.28.243.2:35975)
0812 16:44:46.506989 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:46.506996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:46.507024 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:44:46.507095 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:46.507103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:46.507316 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:46.507449 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:46.508501 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38224: BlockingRecv error: recv got EOF from 127.0.0.1:38224 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":254,"threads_started":1}
W20260812 16:44:46.508746 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1811: Not authorized: Client connection negotiation failed: client connection to 127.28.243.2:35975: server requires authentication, but client does not have Kerberos credentials available
I20260812 16:44:46.508920 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1811
I20260812 16:44:46.536392 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2043
W20260812 16:44:46.540503 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2043: Not authorized: Client connection negotiation failed: client connection to 127.28.243.3:39551: server requires authentication, but client does not have Kerberos credentials available
I20260812 16:44:46.540689 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2043
I20260812 16:44:46.569591 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1405
I20260812 16:44:46.571398 1526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:46.678784 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1405
2026-08-12T16:44:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4204 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:46 dist-test-slave-1vmg krb5kdc[2305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:46 dist-test-slave-1vmg krb5kdc[2305](info): set up 2 sockets
Aug 12 16:44:46 dist-test-slave-1vmg krb5kdc[2305](info): commencing operation
krb5kdc: starting...
W20260812 16:44:47.994529 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:48 dist-test-slave-1vmg krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:48.241973 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:33369
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:48.598059 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:48.598397 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:48.598490 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:48.606135 2329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:48.606278 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:48.606329 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:48.606380 2329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:48.606443 2329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:48.616971 2329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:33369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:48.618353 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:48.619752 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:48.631886 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:48.631863 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:48.632007 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:48.638718 2329 server_base.cc:1034] running on GCE node
I20260812 16:44:48.639683 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:48.641270 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:48.642521 2329 hybrid_clock.cc:648] HybridClock initialized: now 1786553088642469 us; error 40 us; skew 500 ppm
I20260812 16:44:48.642895 2329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:48 dist-test-slave-1vmg krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:48.650628 2329 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:48.653146 2329 webserver.cc:460] Webserver started at http://127.28.243.62:39437/ using document root <none> and password file <none>
I20260812 16:44:48.653976 2329 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:48.654100 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:48.654444 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:48.657794 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "df06b79565fd41219c5f1ac9d6fba52d"
format_stamp: "Formatted at 2026-08-12 16:44:48 on dist-test-slave-1vmg"
server_key: "7bba6ce8b4455554e700a2fdf7ed7bd8"
server_key_iv: "00bec709a39500c86a72cc168db15ab1"
server_key_version: "encryptionkey@0"
I20260812 16:44:48.658738 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "df06b79565fd41219c5f1ac9d6fba52d"
format_stamp: "Formatted at 2026-08-12 16:44:48 on dist-test-slave-1vmg"
server_key: "7bba6ce8b4455554e700a2fdf7ed7bd8"
server_key_iv: "00bec709a39500c86a72cc168db15ab1"
server_key_version: "encryptionkey@0"
I20260812 16:44:48.664587 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 16:44:48.669492 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:48.671160 2329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:44:48.671360 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "df06b79565fd41219c5f1ac9d6fba52d"
format_stamp: "Formatted at 2026-08-12 16:44:48 on dist-test-slave-1vmg"
server_key: "7bba6ce8b4455554e700a2fdf7ed7bd8"
server_key_iv: "00bec709a39500c86a72cc168db15ab1"
server_key_version: "encryptionkey@0"
I20260812 16:44:48.671550 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:48.685930 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:48.693457 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:48.693764 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:48.718076 2329 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:33369
I20260812 16:44:48.718097 2442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:33369 every 8 connection(s)
I20260812 16:44:48.719655 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:48.724094 2444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:48.726528 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2329
I20260812 16:44:48.726869 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:48.728523 29644 external_mini_cluster.cc:1428] Setting key 519046c29e6f7f7ecd2a88d7ddc751f2
I20260812 16:44:48.738597 2444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d: Bootstrap starting.
I20260812 16:44:48.742676 2444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:48.743806 2444 log.cc:824] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:44:48 dist-test-slave-1vmg krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:48.747313 2444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d: No bootstrap required, opened a new log
I20260812 16:44:48.757138 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:48.731097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.28.243.62:33369)
0812 16:44:48.732054 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:48.732069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:48.732129 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:44:48.734096 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:48.734133 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:48.734796 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:48.735329 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:48.736817 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:48.738811 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:48.740207 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:48.741336 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:48.741544 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:48.746318 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:48.746331 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:48.746358 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:48.746422 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:48.751663 (+ 5241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:48.752830 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:48.752857 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:48.752869 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:48.753289 (+ 420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:48.753917 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:48.753926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:48.753929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:48.754720 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:48.755032 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:48.755552 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 16:44:48.756006 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":443,"threads_started":1}
I20260812 16:44:48.756923 2444 raft_consensus.cc:348] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df06b79565fd41219c5f1ac9d6fba52d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 33369 } }
I20260812 16:44:48.757725 2444 raft_consensus.cc:374] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:48.757860 2444 raft_consensus.cc:729] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df06b79565fd41219c5f1ac9d6fba52d, State: Initialized, Role: FOLLOWER
I20260812 16:44:48.758469 2444 consensus_queue.cc:260] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df06b79565fd41219c5f1ac9d6fba52d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 33369 } }
I20260812 16:44:48.758733 2444 raft_consensus.cc:388] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:48.758874 2444 raft_consensus.cc:482] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:48.759087 2444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:48.761297 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:44:48.761700 2444 raft_consensus.cc:504] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df06b79565fd41219c5f1ac9d6fba52d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 33369 } }
I20260812 16:44:48.762209 2444 leader_election.cc:302] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df06b79565fd41219c5f1ac9d6fba52d; no voters:
I20260812 16:44:48.763029 2444 leader_election.cc:288] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:48.763180 2450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:48.765275 2444 sys_catalog.cc:564] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:48.769590 2450 raft_consensus.cc:686] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [term 1 LEADER]: Becoming Leader. State: Replica: df06b79565fd41219c5f1ac9d6fba52d, State: Running, Role: LEADER
I20260812 16:44:48.770249 2450 consensus_queue.cc:237] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df06b79565fd41219c5f1ac9d6fba52d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 33369 } }
I20260812 16:44:48.774660 2451 sys_catalog.cc:455] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df06b79565fd41219c5f1ac9d6fba52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df06b79565fd41219c5f1ac9d6fba52d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 33369 } } }
I20260812 16:44:48.775147 2451 sys_catalog.cc:458] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:48.775732 2452 sys_catalog.cc:455] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df06b79565fd41219c5f1ac9d6fba52d. Latest consensus state: current_term: 1 leader_uuid: "df06b79565fd41219c5f1ac9d6fba52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df06b79565fd41219c5f1ac9d6fba52d" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 33369 } } }
I20260812 16:44:48.776412 2452 sys_catalog.cc:458] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:48.777117 2479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:48.786372 2479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:48.806674 2479 catalog_manager.cc:1347] Generated new cluster ID: 6364a8a97227438287b4e7f6cfccfe76
I20260812 16:44:48.807063 2479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:48.821098 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:48.813274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.28.243.62:33369)
0812 16:44:48.813470 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:48.813478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:48.814090 (+ 612us) server_negotiation.cc:407] Connection header received
0812 16:44:48.814189 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:48.814196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:48.814271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:48.814364 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:48.815489 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:48.816239 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:48.817578 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:48.818031 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:48.818147 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:48.819315 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:48.819320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:48.819324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:44:48.820254 (+ 930us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:48.820329 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:44:48.820354 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:48.820466 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:48.820631 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 16:44:48.820756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:44:48.849184 2479 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:48.850721 2479 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:48.861172 2479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df06b79565fd41219c5f1ac9d6fba52d: Generated new TSK 0
I20260812 16:44:48.862744 2479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:44:48.879663 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2329
I20260812 16:44:48.881296 2436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:48.984897 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2329
2026-08-12T16:44:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2304 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:49 dist-test-slave-1vmg krb5kdc[2505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:49 dist-test-slave-1vmg krb5kdc[2505](info): set up 2 sockets
Aug 12 16:44:49 dist-test-slave-1vmg krb5kdc[2505](info): commencing operation
krb5kdc: starting...
W20260812 16:44:50.300333 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:50 dist-test-slave-1vmg krb5kdc[2505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:50.556742 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45193
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:45193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:50.916991 2527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:50.917326 2527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:50.917419 2527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:50.925264 2527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:50.925407 2527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:50.925455 2527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:50.925514 2527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:50.925559 2527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:50.936540 2527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:45193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:50.937968 2527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:50.939357 2527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:50.954545 2539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:50.954643 2536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:50.954628 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:50.955386 2527 server_base.cc:1034] running on GCE node
I20260812 16:44:50.956373 2527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:50.958292 2527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:50.959645 2527 hybrid_clock.cc:648] HybridClock initialized: now 1786553090959557 us; error 69 us; skew 500 ppm
I20260812 16:44:50.960227 2527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:50 dist-test-slave-1vmg krb5kdc[2505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553090, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:50.964839 2527 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:50.966648 2527 webserver.cc:460] Webserver started at http://127.28.243.62:34561/ using document root <none> and password file <none>
I20260812 16:44:50.967455 2527 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:50.967595 2527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:50.967918 2527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:50.971351 2527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1"
format_stamp: "Formatted at 2026-08-12 16:44:50 on dist-test-slave-1vmg"
server_key: "27177245de3c26e3791fad53189dd340"
server_key_iv: "fff53ca38b205a8b8f7b2898155d1925"
server_key_version: "encryptionkey@0"
I20260812 16:44:50.972199 2527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1"
format_stamp: "Formatted at 2026-08-12 16:44:50 on dist-test-slave-1vmg"
server_key: "27177245de3c26e3791fad53189dd340"
server_key_iv: "fff53ca38b205a8b8f7b2898155d1925"
server_key_version: "encryptionkey@0"
I20260812 16:44:50.978433 2527 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:44:50.982474 2555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:50.983989 2527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:44:50.984198 2527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1"
format_stamp: "Formatted at 2026-08-12 16:44:50 on dist-test-slave-1vmg"
server_key: "27177245de3c26e3791fad53189dd340"
server_key_iv: "fff53ca38b205a8b8f7b2898155d1925"
server_key_version: "encryptionkey@0"
I20260812 16:44:50.984404 2527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:51.001109 2527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:51.005795 2527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:51.006095 2527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:51.026407 2527 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:45193
I20260812 16:44:51.026428 2660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:45193 every 8 connection(s)
I20260812 16:44:51.028090 2527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:51.032598 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2527
I20260812 16:44:51.032765 2662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:51.032846 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:51.033687 29644 external_mini_cluster.cc:1428] Setting key 0d3d586ff4160cc95335877932b7f96a
I20260812 16:44:51.046499 2662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1: Bootstrap starting.
Aug 12 16:44:51 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:51.050665 2662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:51.051869 2662 log.cc:824] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:51.056319 2662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1: No bootstrap required, opened a new log
I20260812 16:44:51.059813 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:51.036010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.28.243.62:45193)
0812 16:44:51.037021 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:51.037037 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:51.037095 (+ 58us) server_negotiation.cc:407] Connection header received
0812 16:44:51.039568 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:51.039596 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:51.040112 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:51.040552 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:51.042062 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:51.043501 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:51.045371 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:51.046362 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:51.046610 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:51.051333 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:51.051343 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:51.051362 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:51.051411 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:51.054890 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:51.055753 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:51.055761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:51.055769 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:51.055955 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:51.056499 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:51.056504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:51.056507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:51.057169 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:51.057510 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:51.058126 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 16:44:51.058632 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":407,"threads_started":1}
I20260812 16:44:51.065286 2662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45193 } }
I20260812 16:44:51.065701 2662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:51.065793 2662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9a416511cc42dd8c2d6ac88b1cd8b1, State: Initialized, Role: FOLLOWER
I20260812 16:44:51.066740 2662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45193 } }
I20260812 16:44:51.067193 2662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:51.067495 2662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:51.067727 2662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:51.070569 2662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45193 } }
I20260812 16:44:51.071367 2662 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9a416511cc42dd8c2d6ac88b1cd8b1; no voters:
I20260812 16:44:51.072079 2662 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:51.072292 2672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:51.073376 2672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9a416511cc42dd8c2d6ac88b1cd8b1, State: Running, Role: LEADER
I20260812 16:44:51.074002 2672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45193 } }
I20260812 16:44:51.074555 2662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:51.077123 2673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45193 } } }
I20260812 16:44:51.077533 2673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:51.077157 2676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9a416511cc42dd8c2d6ac88b1cd8b1. Latest consensus state: current_term: 1 leader_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a416511cc42dd8c2d6ac88b1cd8b1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45193 } } }
I20260812 16:44:51.078287 2676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:51.084543 2683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:51.090622 2683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:51.100955 2683 catalog_manager.cc:1347] Generated new cluster ID: 80f6dc514d2c4f83921a194f7217680b
I20260812 16:44:51.101111 2683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:51.125690 2683 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:51.127244 2683 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:51.137100 2683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9a416511cc42dd8c2d6ac88b1cd8b1: Generated new TSK 0
I20260812 16:44:51.138187 2683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:51.207609 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45193
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:51.600680 2714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:51.601002 2714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:51.601148 2714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:51.611712 2714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:51.612119 2714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:51.612351 2714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:51.624297 2714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:51.626344 2714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:51.628170 2714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:51.645315 2725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:51.645542 2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:51.645581 2722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:51.646590 2714 server_base.cc:1034] running on GCE node
I20260812 16:44:51.647734 2714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:51.649112 2714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:51.651163 2714 hybrid_clock.cc:648] HybridClock initialized: now 1786553091651088 us; error 55 us; skew 500 ppm
I20260812 16:44:51.651698 2714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:51 dist-test-slave-1vmg krb5kdc[2505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553091, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:51.655632 2714 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:51.657281 2714 webserver.cc:460] Webserver started at http://127.28.243.1:38441/ using document root <none> and password file <none>
I20260812 16:44:51.657982 2714 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:51.658089 2714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:51.658375 2714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:51.661580 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "4601272f2a86458cbc75fa779c995a0d"
format_stamp: "Formatted at 2026-08-12 16:44:51 on dist-test-slave-1vmg"
server_key: "4149828585e1658e83d0c1cb917e1497"
server_key_iv: "736bb87dd0a2c7658abb6a40841d274b"
server_key_version: "encryptionkey@0"
I20260812 16:44:51.662380 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "4601272f2a86458cbc75fa779c995a0d"
format_stamp: "Formatted at 2026-08-12 16:44:51 on dist-test-slave-1vmg"
server_key: "4149828585e1658e83d0c1cb917e1497"
server_key_iv: "736bb87dd0a2c7658abb6a40841d274b"
server_key_version: "encryptionkey@0"
I20260812 16:44:51.668474 2714 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 16:44:51.672595 2741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:51.674397 2714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:44:51.674573 2714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "4601272f2a86458cbc75fa779c995a0d"
format_stamp: "Formatted at 2026-08-12 16:44:51 on dist-test-slave-1vmg"
server_key: "4149828585e1658e83d0c1cb917e1497"
server_key_iv: "736bb87dd0a2c7658abb6a40841d274b"
server_key_version: "encryptionkey@0"
I20260812 16:44:51.674798 2714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:51.700632 2714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:51.706080 2714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:51.706429 2714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:51.707592 2714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:51.709077 2714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:51.709167 2714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:51.709290 2714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:51.709347 2714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:51.742267 2714 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:36381
I20260812 16:44:51.742326 2978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:36381 every 8 connection(s)
I20260812 16:44:51.743781 2714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:51.749111 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2714
I20260812 16:44:51.749341 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:51.750018 29644 external_mini_cluster.cc:1428] Setting key 6b63a8afafcb4fa4a9faebe1bb543ebd
Aug 12 16:44:51 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553091, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:51.768244 2982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:51.747146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:34183 (local address 127.28.243.62:45193)
0812 16:44:51.748056 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:51.748075 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:51.749674 (+ 1599us) server_negotiation.cc:407] Connection header received
0812 16:44:51.752583 (+ 2909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:51.752589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:51.752673 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:51.752891 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:51.754878 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:51.755644 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:51.757303 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:51.757698 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:51.757826 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:51.761607 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:51.761614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:51.761618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:51.761658 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:51.764889 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:51.765781 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:51.765787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:51.765791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:51.765951 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:51.766364 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:51.766368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:51.766371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:51.766698 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:51.766842 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:51.767639 (+ 797us) server_negotiation.cc:299] Negotiation successful
0812 16:44:51.767903 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":506,"threads_started":1}
I20260812 16:44:51.769131 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:51.747510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45193 (local address 127.28.243.1:34183)
0812 16:44:51.748914 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:51.749102 (+ 188us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:51.752070 (+ 2968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:51.753291 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:51.753302 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:51.754024 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:51.754665 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:51.754686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:51.755815 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:51.755819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:51.757120 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:51.757135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:51.757963 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:51.757971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:51.758217 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:51.758903 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:51.758956 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:51.761308 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:51.765116 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:51.765125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:51.765141 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:51.765633 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:51.766051 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:51.766054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:51.766056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:51.766259 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:51.766818 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:51.766824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:51.767253 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 16:44:51.767501 (+ 248us) client_negotiation.cc:240] Negotiation successful
0812 16:44:51.767956 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":841,"threads_started":1}
I20260812 16:44:51.770694 2979 heartbeater.cc:347] Connected to a master server at 127.28.243.62:45193
I20260812 16:44:51.771030 2979 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:51.771783 2979 heartbeater.cc:511] Master 127.28.243.62:45193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:44:51.773958 2592 ts_manager.cc:194] Registered new tserver with Master: 4601272f2a86458cbc75fa779c995a0d (127.28.243.1:36381)
I20260812 16:44:51.775590 2592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:34183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:51.812542 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45193
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:52.204897 2990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:52.205271 2990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:52.205381 2990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:52.213093 2990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:52.213248 2990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:52.213367 2990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:52.223668 2990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:52.225245 2990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:52.227573 2990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:52.247666 3001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:52.247742 3004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:52.247725 3002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:52.248225 2990 server_base.cc:1034] running on GCE node
I20260812 16:44:52.249658 2990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:52.251220 2990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:52.252504 2990 hybrid_clock.cc:648] HybridClock initialized: now 1786553092252442 us; error 63 us; skew 500 ppm
I20260812 16:44:52.253214 2990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:52 dist-test-slave-1vmg krb5kdc[2505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:52.256968 2990 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:52.258808 2990 webserver.cc:460] Webserver started at http://127.28.243.2:43595/ using document root <none> and password file <none>
I20260812 16:44:52.259589 2990 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:52.259742 2990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:52.260079 2990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:52.263453 2990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "dbe53b92f7434f3dab794351c8d7ba59"
format_stamp: "Formatted at 2026-08-12 16:44:52 on dist-test-slave-1vmg"
server_key: "94f063b38515d0ab1a21240c5243d797"
server_key_iv: "c73c85ba70f646dec03677bd64a9bb43"
server_key_version: "encryptionkey@0"
I20260812 16:44:52.264324 2990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "dbe53b92f7434f3dab794351c8d7ba59"
format_stamp: "Formatted at 2026-08-12 16:44:52 on dist-test-slave-1vmg"
server_key: "94f063b38515d0ab1a21240c5243d797"
server_key_iv: "c73c85ba70f646dec03677bd64a9bb43"
server_key_version: "encryptionkey@0"
I20260812 16:44:52.272431 2990 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260812 16:44:52.277714 3021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:52.279546 2990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:44:52.279748 2990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "dbe53b92f7434f3dab794351c8d7ba59"
format_stamp: "Formatted at 2026-08-12 16:44:52 on dist-test-slave-1vmg"
server_key: "94f063b38515d0ab1a21240c5243d797"
server_key_iv: "c73c85ba70f646dec03677bd64a9bb43"
server_key_version: "encryptionkey@0"
I20260812 16:44:52.279979 2990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:52.297660 2990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:52.301752 2990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:52.302042 2990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:52.303071 2990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:52.305442 2990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:52.305539 2990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:52.305670 2990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:52.305733 2990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:52.345274 2990 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:40913
I20260812 16:44:52.345332 3233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:40913 every 8 connection(s)
I20260812 16:44:52.347155 2990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:52.352315 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2990
I20260812 16:44:52.352511 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:52.353042 29644 external_mini_cluster.cc:1428] Setting key beda4999af3ffa81300b0e267869fdbd
Aug 12 16:44:52 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:52.373543 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:52.349924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:54103 (local address 127.28.243.62:45193)
0812 16:44:52.350378 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:52.350382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:52.351448 (+ 1066us) server_negotiation.cc:407] Connection header received
0812 16:44:52.353930 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:52.353936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:52.354029 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:52.354134 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:52.356236 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.357014 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.359459 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.359950 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.360064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:52.365149 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:52.365160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:52.365166 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:52.365240 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:52.369044 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:52.369990 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:52.369996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:52.369999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:52.370182 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:52.371060 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:52.371067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:52.371071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:52.371652 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:52.371855 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:52.372755 (+ 900us) server_negotiation.cc:299] Negotiation successful
0812 16:44:52.373127 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260812 16:44:52.374281 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:52.350497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45193 (local address 127.28.243.2:54103)
0812 16:44:52.351273 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:52.351320 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:52.353592 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:52.354420 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:52.354429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:52.355068 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:52.355978 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:52.356010 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.357216 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.357222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:52.359228 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:52.359248 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.360226 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.360234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:52.360541 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:52.361371 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:52.361403 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:52.364509 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:52.369276 (+ 4767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:52.369285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:52.369301 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:52.369803 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:52.370496 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:52.370500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:52.370503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:52.370823 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:52.371863 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:52.371870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:52.372298 (+ 428us) client_negotiation.cc:769] Sending connection context
0812 16:44:52.372566 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 16:44:52.373031 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":301,"threads_started":1}
I20260812 16:44:52.376308 3234 heartbeater.cc:347] Connected to a master server at 127.28.243.62:45193
I20260812 16:44:52.376649 3234 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:52.377467 3234 heartbeater.cc:511] Master 127.28.243.62:45193 requested a full tablet report, sending...
I20260812 16:44:52.379356 2592 ts_manager.cc:194] Registered new tserver with Master: dbe53b92f7434f3dab794351c8d7ba59 (127.28.243.2:40913)
I20260812 16:44:52.380601 2592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:54103
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:52.416110 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45193
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:52.779034 2979 heartbeater.cc:503] Master 127.28.243.62:45193 was elected leader, sending a full tablet report...
W20260812 16:44:52.782140 3242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:52.782433 3242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:52.782570 3242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:52.790042 3242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:52.790182 3242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:52.790338 3242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:52.800623 3242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:52.802127 3242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:52.803565 3242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:52.818295 3252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:52.818320 3253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:52.818493 3256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:52.819187 3242 server_base.cc:1034] running on GCE node
I20260812 16:44:52.819901 3242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:52.820945 3242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:52.822225 3242 hybrid_clock.cc:648] HybridClock initialized: now 1786553092822172 us; error 41 us; skew 500 ppm
I20260812 16:44:52.822582 3242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:52 dist-test-slave-1vmg krb5kdc[2505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:52.826040 3242 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:52.827703 3242 webserver.cc:460] Webserver started at http://127.28.243.3:36323/ using document root <none> and password file <none>
I20260812 16:44:52.828401 3242 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:52.828505 3242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:52.828773 3242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:52.831849 3242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "f171d0ca11fa48b086a6183c27fef8f0"
format_stamp: "Formatted at 2026-08-12 16:44:52 on dist-test-slave-1vmg"
server_key: "40324e3c80a486de0232cd9ea406fa4c"
server_key_iv: "57303a15c64e153da5c8c4116df694cb"
server_key_version: "encryptionkey@0"
I20260812 16:44:52.832690 3242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "f171d0ca11fa48b086a6183c27fef8f0"
format_stamp: "Formatted at 2026-08-12 16:44:52 on dist-test-slave-1vmg"
server_key: "40324e3c80a486de0232cd9ea406fa4c"
server_key_iv: "57303a15c64e153da5c8c4116df694cb"
server_key_version: "encryptionkey@0"
I20260812 16:44:52.838198 3242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 16:44:52.841985 3266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:52.843516 3242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:44:52.843708 3242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "f171d0ca11fa48b086a6183c27fef8f0"
format_stamp: "Formatted at 2026-08-12 16:44:52 on dist-test-slave-1vmg"
server_key: "40324e3c80a486de0232cd9ea406fa4c"
server_key_iv: "57303a15c64e153da5c8c4116df694cb"
server_key_version: "encryptionkey@0"
I20260812 16:44:52.843916 3242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:52.866313 3242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:52.873246 3242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:52.873674 3242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:52.875366 3242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:52.877151 3242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:52.877252 3242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:52.877389 3242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:52.877444 3242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:52.915658 3242 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:34449
I20260812 16:44:52.915762 3470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:34449 every 8 connection(s)
I20260812 16:44:52.917554 3242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:52.924453 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3242
I20260812 16:44:52.924685 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:52.925163 29644 external_mini_cluster.cc:1428] Setting key 6a186416aa8eacf42818e7b48e2cd066
Aug 12 16:44:52 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:52.944268 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:52.920833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:55387 (local address 127.28.243.62:45193)
0812 16:44:52.921377 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:52.921383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:52.922362 (+ 979us) server_negotiation.cc:407] Connection header received
0812 16:44:52.924431 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:52.924439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:52.924542 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:52.924677 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:52.926951 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.928053 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.930726 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.931224 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.931360 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:52.936544 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:52.936551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:52.936556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:52.936617 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:52.939489 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:52.940532 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:52.940538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:52.940541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:52.940706 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:52.941253 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:52.941259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:52.941262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:52.941654 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:52.941805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:52.943730 (+ 1925us) server_negotiation.cc:299] Negotiation successful
0812 16:44:52.943949 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":242,"threads_started":1}
I20260812 16:44:52.945374 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:52.921338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45193 (local address 127.28.243.3:55387)
0812 16:44:52.922180 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:52.922230 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:52.924060 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:52.924965 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:52.924975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:52.925620 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:52.926710 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:52.926749 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.928363 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.928380 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:52.930427 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:52.930454 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:52.931576 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:52.931583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:52.931862 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:52.932719 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:52.932765 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:52.936133 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:52.939780 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:52.939792 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:52.939810 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:52.940404 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:52.940879 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:52.940885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:52.940888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:52.941129 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:52.942009 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:52.942023 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:52.942769 (+ 746us) client_negotiation.cc:769] Sending connection context
0812 16:44:52.943364 (+ 595us) client_negotiation.cc:240] Negotiation successful
0812 16:44:52.943901 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":326,"threads_started":1}
I20260812 16:44:52.947607 3471 heartbeater.cc:347] Connected to a master server at 127.28.243.62:45193
I20260812 16:44:52.947973 3471 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:52.948869 3471 heartbeater.cc:511] Master 127.28.243.62:45193 requested a full tablet report, sending...
I20260812 16:44:52.950639 2592 ts_manager.cc:194] Registered new tserver with Master: f171d0ca11fa48b086a6183c27fef8f0 (127.28.243.3:34449)
I20260812 16:44:52.951560 2592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:55387
I20260812 16:44:52.955601 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:44:52 dist-test-slave-1vmg krb5kdc[2505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:53.017776 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:52.999203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.28.243.62:45193)
0812 16:44:52.999371 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:52.999376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:52.999875 (+ 499us) server_negotiation.cc:407] Connection header received
0812 16:44:53.000114 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:53.000119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:53.000193 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:53.000295 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:53.001833 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.002635 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.004343 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.004790 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.004903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:53.009085 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:53.009091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:53.009095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:53.009136 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:53.014156 (+ 5020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.015271 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.015277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.015280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.015463 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.016246 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.016250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.016253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.016650 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:53.016890 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:53.017313 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 16:44:53.017526 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 16:44:53.018623 2592 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40036
I20260812 16:44:53.021158 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2714
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:53.043325 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:53.022509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.28.243.1:36381)
0812 16:44:53.023160 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:53.023170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:53.023198 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:44:53.023259 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:53.023266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:53.023484 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:53.023624 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:53.025113 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.026560 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.028781 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.030403 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.030616 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:53.036411 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:53.036435 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:53.036457 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:53.036514 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:53.040008 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.040925 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.040933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.040935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.041090 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.041502 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.041506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.041509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.041926 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:53.042235 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:53.042605 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 16:44:53.042943 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":302,"threads_started":1}
W20260812 16:44:53.044857 2953 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40898
W20260812 16:44:53.046694 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2714: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:53.047216 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2714
I20260812 16:44:53.074676 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2990
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:44:53.092753 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:53.075781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.28.243.2:40913)
0812 16:44:53.076365 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:53.076372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:53.076401 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:44:53.076464 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:53.076471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:53.076694 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:53.076829 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:53.078141 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.079296 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.080797 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.081437 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.081557 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:53.086615 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:53.086628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:53.086645 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:53.086688 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:53.089836 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.090545 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.090551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.090554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.090681 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.091052 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.091056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.091059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.091445 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:53.091793 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:53.092114 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 16:44:53.092385 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":300,"threads_started":1}
W20260812 16:44:53.094050 3227 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55858
W20260812 16:44:53.095342 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:53.095584 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2990
I20260812 16:44:53.122366 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3242
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[2505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:44:53.141233 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:53.123492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.28.243.3:34449)
0812 16:44:53.124431 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:53.124438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:53.124469 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:44:53.124534 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:53.124541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:53.124760 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:53.124904 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:53.126395 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.127677 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.129114 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:53.130175 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:53.130299 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:53.134871 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:53.134887 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:53.134904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:53.135043 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:53.138009 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.138833 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.138841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.138844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.139068 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:53.139521 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:53.139526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:53.139528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:53.139969 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:53.140255 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:53.140634 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 16:44:53.140885 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":581,"threads_started":1}
W20260812 16:44:53.142678 3462 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52868
W20260812 16:44:53.143997 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:44:53.144186 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3242
I20260812 16:44:53.190475 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2527
I20260812 16:44:53.191808 2644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:53.304231 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2527
2026-08-12T16:44:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4326 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[3517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[3517](info): set up 2 sockets
Aug 12 16:44:53 dist-test-slave-1vmg krb5kdc[3517](info): commencing operation
krb5kdc: starting...
W20260812 16:44:54.629737 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.008s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:54 dist-test-slave-1vmg krb5kdc[3517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:54.881067 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:34161
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:55.284382 3544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:55.284723 3544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:55.284860 3544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:55.294154 3544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:55.294334 3544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:55.294418 3544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:55.294507 3544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:55.294566 3544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:55.305933 3544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:55.308111 3544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:55.311188 3544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:55.327279 3553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:55.327328 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:55.327577 3560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:55.328667 3544 server_base.cc:1034] running on GCE node
I20260812 16:44:55.329737 3544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:55.331746 3544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:55.333060 3544 hybrid_clock.cc:648] HybridClock initialized: now 1786553095332966 us; error 78 us; skew 500 ppm
I20260812 16:44:55.333446 3544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:55 dist-test-slave-1vmg krb5kdc[3517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553095, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:55.342316 3544 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:55.344041 3544 webserver.cc:460] Webserver started at http://127.28.243.62:37959/ using document root <none> and password file <none>
I20260812 16:44:55.344791 3544 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:55.344910 3544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:55.345235 3544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:55.349310 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "064525634c014fba93b381bfc65ee723"
format_stamp: "Formatted at 2026-08-12 16:44:55 on dist-test-slave-1vmg"
server_key: "e44294ee8034735f4469b3054c446937"
server_key_iv: "a18fd3951ee3d99a4cfd243fdd6292b6"
server_key_version: "encryptionkey@0"
I20260812 16:44:55.350261 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "064525634c014fba93b381bfc65ee723"
format_stamp: "Formatted at 2026-08-12 16:44:55 on dist-test-slave-1vmg"
server_key: "e44294ee8034735f4469b3054c446937"
server_key_iv: "a18fd3951ee3d99a4cfd243fdd6292b6"
server_key_version: "encryptionkey@0"
I20260812 16:44:55.356130 3544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 16:44:55.359992 3572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:55.361580 3544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:44:55.361789 3544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "064525634c014fba93b381bfc65ee723"
format_stamp: "Formatted at 2026-08-12 16:44:55 on dist-test-slave-1vmg"
server_key: "e44294ee8034735f4469b3054c446937"
server_key_iv: "a18fd3951ee3d99a4cfd243fdd6292b6"
server_key_version: "encryptionkey@0"
I20260812 16:44:55.362003 3544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:55.373574 3544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:55.379302 3544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:55.379679 3544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:55.400627 3544 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:34161
I20260812 16:44:55.400658 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:34161 every 8 connection(s)
I20260812 16:44:55.402698 3544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:55.408151 3663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:55.410624 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3544
I20260812 16:44:55.410889 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:55.411595 29644 external_mini_cluster.cc:1428] Setting key ce68bec4aa1e59756e43992f666e431d
I20260812 16:44:55.424764 3663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723: Bootstrap starting.
Aug 12 16:44:55 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:55.429030 3663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:55.430466 3663 log.cc:824] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:55.434827 3663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723: No bootstrap required, opened a new log
I20260812 16:44:55.439843 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:55.413891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.28.243.62:34161)
0812 16:44:55.414761 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:55.414773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:55.414819 (+ 46us) server_negotiation.cc:407] Connection header received
0812 16:44:55.416529 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:55.416555 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:55.417294 (+ 739us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:55.417716 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:55.419798 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:55.421276 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:55.423236 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:55.424026 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:55.424185 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:55.429194 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:55.429215 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:55.429239 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:55.429295 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:55.433797 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:55.434701 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:55.434710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:55.434720 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:55.435026 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:55.435644 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:55.435654 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:55.435658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:55.436633 (+ 975us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:55.436971 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:55.437706 (+ 735us) server_negotiation.cc:299] Negotiation successful
0812 16:44:55.438267 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":382,"threads_started":1}
I20260812 16:44:55.445135 3663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064525634c014fba93b381bfc65ee723" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34161 } }
I20260812 16:44:55.445919 3663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:44:55.446151 3663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064525634c014fba93b381bfc65ee723, State: Initialized, Role: FOLLOWER
I20260812 16:44:55.447137 3663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064525634c014fba93b381bfc65ee723" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34161 } }
I20260812 16:44:55.447438 3663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:44:55.447576 3663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:44:55.447729 3663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:55.450588 3663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064525634c014fba93b381bfc65ee723" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34161 } }
I20260812 16:44:55.451179 3663 leader_election.cc:302] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 064525634c014fba93b381bfc65ee723; no voters:
I20260812 16:44:55.451766 3663 leader_election.cc:288] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:44:55.451912 3677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:55.453034 3677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [term 1 LEADER]: Becoming Leader. State: Replica: 064525634c014fba93b381bfc65ee723, State: Running, Role: LEADER
I20260812 16:44:55.453613 3677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064525634c014fba93b381bfc65ee723" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34161 } }
I20260812 16:44:55.454003 3663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:55.456511 3680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 064525634c014fba93b381bfc65ee723. Latest consensus state: current_term: 1 leader_uuid: "064525634c014fba93b381bfc65ee723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064525634c014fba93b381bfc65ee723" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34161 } } }
I20260812 16:44:55.456866 3680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:55.457322 3678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "064525634c014fba93b381bfc65ee723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064525634c014fba93b381bfc65ee723" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34161 } } }
I20260812 16:44:55.457633 3678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723 [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:55.457914 3692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:44:55.463352 3692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:55.473477 3692 catalog_manager.cc:1347] Generated new cluster ID: bd624a31169c4cefa1da7bbbcc5a77f5
I20260812 16:44:55.473621 3692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:55.492738 3692 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:55.494642 3692 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:55.511220 3692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 064525634c014fba93b381bfc65ee723: Generated new TSK 0
I20260812 16:44:55.512363 3692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:44:55.587203 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34161
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:55.946907 3718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:55.947288 3718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:55.947391 3718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:55.954964 3718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:55.955091 3718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:55.955230 3718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:44:55.966101 3718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:55.967527 3718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:55.968830 3718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:55.982333 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:55.982282 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:55.982358 3741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:55.983080 3718 server_base.cc:1034] running on GCE node
I20260812 16:44:55.983769 3718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:55.984812 3718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:55.986176 3718 hybrid_clock.cc:648] HybridClock initialized: now 1786553095986110 us; error 50 us; skew 500 ppm
I20260812 16:44:55.986527 3718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:55 dist-test-slave-1vmg krb5kdc[3517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553095, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:55.989924 3718 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:44:55.991634 3718 webserver.cc:460] Webserver started at http://127.28.243.1:41833/ using document root <none> and password file <none>
I20260812 16:44:55.992352 3718 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:55.992475 3718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:55.992779 3718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:55.995828 3718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "7ed401a64c694bbbad4bc1d5cd3c8527"
format_stamp: "Formatted at 2026-08-12 16:44:55 on dist-test-slave-1vmg"
server_key: "d2615b9b0dcdd4efdc27e4b54ec3629b"
server_key_iv: "a77ddd86ed3a1edf6090bb13f84fd02f"
server_key_version: "encryptionkey@0"
I20260812 16:44:55.996657 3718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "7ed401a64c694bbbad4bc1d5cd3c8527"
format_stamp: "Formatted at 2026-08-12 16:44:55 on dist-test-slave-1vmg"
server_key: "d2615b9b0dcdd4efdc27e4b54ec3629b"
server_key_iv: "a77ddd86ed3a1edf6090bb13f84fd02f"
server_key_version: "encryptionkey@0"
I20260812 16:44:56.002187 3718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:44:56.005777 3751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:56.007197 3718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:44:56.007372 3718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "7ed401a64c694bbbad4bc1d5cd3c8527"
format_stamp: "Formatted at 2026-08-12 16:44:55 on dist-test-slave-1vmg"
server_key: "d2615b9b0dcdd4efdc27e4b54ec3629b"
server_key_iv: "a77ddd86ed3a1edf6090bb13f84fd02f"
server_key_version: "encryptionkey@0"
I20260812 16:44:56.007567 3718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:56.027645 3718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:56.032158 3718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:56.032454 3718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:56.033555 3718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:56.035157 3718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:56.035246 3718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:56.035383 3718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:56.035439 3718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:56.070611 3718 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:36293
I20260812 16:44:56.070736 3939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:36293 every 8 connection(s)
I20260812 16:44:56.071990 3718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:44:56.073537 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3718
I20260812 16:44:56.073798 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:44:56.074640 29644 external_mini_cluster.cc:1428] Setting key f84b71b127e7fec5f60dce9f64e948b1
Aug 12 16:44:56 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553095, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:44:56.099677 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:56.075066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:54295 (local address 127.28.243.62:34161)
0812 16:44:56.075576 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:56.075582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:56.076859 (+ 1277us) server_negotiation.cc:407] Connection header received
0812 16:44:56.081618 (+ 4759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:56.081625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:56.081719 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:56.081825 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:56.084113 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.084860 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.086793 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.087431 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.087578 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:56.091849 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:56.091859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:56.091866 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:56.091921 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:56.095568 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:56.096544 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:56.096551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:56.096554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:56.096729 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:56.097261 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:56.097266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:56.097270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:56.097643 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:56.097816 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:56.099022 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0812 16:44:56.099285 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
I20260812 16:44:56.100833 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:56.075737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34161 (local address 127.28.243.1:54295)
0812 16:44:56.076687 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:56.076751 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:56.081209 (+ 4458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:56.082146 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:56.082158 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:56.082998 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:56.083843 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:56.083871 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.085108 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.085114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:56.086586 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:56.086602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.087721 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.087727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:56.087950 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:56.088643 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:56.088673 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:56.091521 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:56.095801 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:56.095813 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:56.095830 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:56.096376 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:56.096886 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:56.096890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:56.096892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:56.097119 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:56.098123 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:56.098131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:56.098542 (+ 411us) client_negotiation.cc:769] Sending connection context
0812 16:44:56.098810 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 16:44:56.099452 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":407,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
I20260812 16:44:56.102711 3941 heartbeater.cc:347] Connected to a master server at 127.28.243.62:34161
I20260812 16:44:56.103065 3941 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:56.103850 3941 heartbeater.cc:511] Master 127.28.243.62:34161 requested a full tablet report, sending...
I20260812 16:44:56.106057 3605 ts_manager.cc:194] Registered new tserver with Master: 7ed401a64c694bbbad4bc1d5cd3c8527 (127.28.243.1:36293)
I20260812 16:44:56.107705 3605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:54295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:44:56.130661 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34161
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:56.495586 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:56.495899 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:56.496027 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:56.503538 3951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:56.503672 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:56.503839 3951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:44:56.514330 3951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:56.515815 3951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:56.517215 3951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:56.531599 3957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:56.531481 3956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:56.531503 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:56.532099 3951 server_base.cc:1034] running on GCE node
I20260812 16:44:56.532858 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:56.533939 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:56.535300 3951 hybrid_clock.cc:648] HybridClock initialized: now 1786553096535175 us; error 49 us; skew 500 ppm
I20260812 16:44:56.535879 3951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:56 dist-test-slave-1vmg krb5kdc[3517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:56.539988 3951 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:44:56.541690 3951 webserver.cc:460] Webserver started at http://127.28.243.2:43461/ using document root <none> and password file <none>
I20260812 16:44:56.542464 3951 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:56.542603 3951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:56.542972 3951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:56.546288 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "6e00f9a06edf449ebb69b8ac24953147"
format_stamp: "Formatted at 2026-08-12 16:44:56 on dist-test-slave-1vmg"
server_key: "042a1dcab680556bc64c9326b10f5d57"
server_key_iv: "bb745706d2d36927adf0f4f8d471c2c3"
server_key_version: "encryptionkey@0"
I20260812 16:44:56.547202 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "6e00f9a06edf449ebb69b8ac24953147"
format_stamp: "Formatted at 2026-08-12 16:44:56 on dist-test-slave-1vmg"
server_key: "042a1dcab680556bc64c9326b10f5d57"
server_key_iv: "bb745706d2d36927adf0f4f8d471c2c3"
server_key_version: "encryptionkey@0"
I20260812 16:44:56.553318 3951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:44:56.557662 3977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:56.559980 3951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:44:56.560294 3951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "6e00f9a06edf449ebb69b8ac24953147"
format_stamp: "Formatted at 2026-08-12 16:44:56 on dist-test-slave-1vmg"
server_key: "042a1dcab680556bc64c9326b10f5d57"
server_key_iv: "bb745706d2d36927adf0f4f8d471c2c3"
server_key_version: "encryptionkey@0"
I20260812 16:44:56.560586 3951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:56.580319 3951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:56.584878 3951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:56.585168 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:56.586336 3951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:56.587924 3951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:56.588003 3951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:56.588097 3951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:56.588146 3951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:56.619920 3951 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:46707
I20260812 16:44:56.619961 4163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:46707 every 8 connection(s)
I20260812 16:44:56.621201 3951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:44:56.627908 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3951
I20260812 16:44:56.628141 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:44:56.628690 29644 external_mini_cluster.cc:1428] Setting key 2e0037e09caa7f41ec66b90c9b25777d
Aug 12 16:44:56 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:56.648562 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:56.623871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:36201 (local address 127.28.243.62:34161)
0812 16:44:56.624763 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:56.624769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:56.626057 (+ 1288us) server_negotiation.cc:407] Connection header received
0812 16:44:56.628930 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:56.628938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:56.629102 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:56.629212 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:56.631212 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.631929 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.634314 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.634974 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.635087 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:56.640514 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:56.640521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:56.640525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:56.640577 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:56.643772 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:56.644763 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:56.644768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:56.644771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:56.644913 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:56.645459 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:56.645464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:56.645468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:56.645784 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:56.645935 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:56.647694 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0812 16:44:56.648167 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":578,"threads_started":1}
I20260812 16:44:56.649784 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:56.624887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34161 (local address 127.28.243.2:36201)
0812 16:44:56.625818 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:56.625882 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:56.628494 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:56.629498 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:56.629511 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:56.630203 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:56.631014 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:56.631044 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.632122 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.632128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:56.634111 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:56.634131 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:56.635231 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:56.635238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:56.635536 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:56.636313 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:56.636350 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:56.640086 (+ 3736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:56.643999 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:56.644012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:56.644030 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:56.644596 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:56.645033 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:56.645038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:56.645042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:56.645266 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:56.645928 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:56.645938 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:56.646731 (+ 793us) client_negotiation.cc:769] Sending connection context
0812 16:44:56.647444 (+ 713us) client_negotiation.cc:240] Negotiation successful
0812 16:44:56.648336 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":321,"threads_started":1}
I20260812 16:44:56.651523 4164 heartbeater.cc:347] Connected to a master server at 127.28.243.62:34161
I20260812 16:44:56.651840 4164 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:56.652593 4164 heartbeater.cc:511] Master 127.28.243.62:34161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
I20260812 16:44:56.654184 3605 ts_manager.cc:194] Registered new tserver with Master: 6e00f9a06edf449ebb69b8ac24953147 (127.28.243.2:46707)
I20260812 16:44:56.655200 3605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:36201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:44:56.686290 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34161
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:57.049010 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:57.049355 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:57.049515 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:57.057631 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:57.057780 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:57.057900 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:44:57.069628 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:57.071121 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:57.072467 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:57.086289 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:57.086349 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:57.086377 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:57.086957 4171 server_base.cc:1034] running on GCE node
I20260812 16:44:57.087651 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:57.088727 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:57.090057 4171 hybrid_clock.cc:648] HybridClock initialized: now 1786553097089962 us; error 49 us; skew 500 ppm
I20260812 16:44:57.090662 4171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:57.095037 4171 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:44:57.096920 4171 webserver.cc:460] Webserver started at http://127.28.243.3:34095/ using document root <none> and password file <none>
I20260812 16:44:57.097674 4171 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:57.097790 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:57.098234 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:57.101562 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "96297c13a37848869d3195390613edb1"
format_stamp: "Formatted at 2026-08-12 16:44:57 on dist-test-slave-1vmg"
server_key: "b9842a94246dd387a934ba206b02aaca"
server_key_iv: "8b5a759c758568c334d92f1dd411206b"
server_key_version: "encryptionkey@0"
I20260812 16:44:57.102397 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "96297c13a37848869d3195390613edb1"
format_stamp: "Formatted at 2026-08-12 16:44:57 on dist-test-slave-1vmg"
server_key: "b9842a94246dd387a934ba206b02aaca"
server_key_iv: "8b5a759c758568c334d92f1dd411206b"
server_key_version: "encryptionkey@0"
I20260812 16:44:57.108484 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 16:44:57.110893 3941 heartbeater.cc:503] Master 127.28.243.62:34161 was elected leader, sending a full tablet report...
I20260812 16:44:57.113432 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:57.115270 4171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:44:57.115481 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "96297c13a37848869d3195390613edb1"
format_stamp: "Formatted at 2026-08-12 16:44:57 on dist-test-slave-1vmg"
server_key: "b9842a94246dd387a934ba206b02aaca"
server_key_iv: "8b5a759c758568c334d92f1dd411206b"
server_key_version: "encryptionkey@0"
I20260812 16:44:57.115712 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:57.149006 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:57.154244 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:57.154565 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:57.155774 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:44:57.157263 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:44:57.157352 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:57.157485 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:44:57.157552 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:57.194952 4372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:40147 every 8 connection(s)
I20260812 16:44:57.195047 4171 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:40147
I20260812 16:44:57.196413 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:44:57.203437 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4171
I20260812 16:44:57.203650 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:44:57.204322 29644 external_mini_cluster.cc:1428] Setting key 93ae00be0e47f9ad831e900a412880e0
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:57.222642 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.199572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:60925 (local address 127.28.243.62:34161)
0812 16:44:57.200080 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:57.200086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:57.201215 (+ 1129us) server_negotiation.cc:407] Connection header received
0812 16:44:57.203267 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:57.203273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:57.203394 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:57.203513 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:57.205953 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.207075 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.209292 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.210421 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.210550 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.215019 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:57.215026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:57.215031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:57.215082 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:57.218060 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.219659 (+ 1599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.219666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.219670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.219908 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.220614 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.220618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.220643 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.221073 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:57.221234 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:57.222118 (+ 884us) server_negotiation.cc:299] Negotiation successful
0812 16:44:57.222338 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":225,"threads_started":1}
I20260812 16:44:57.223771 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.200237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34161 (local address 127.28.243.3:60925)
0812 16:44:57.201032 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:57.201074 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:57.202816 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:57.203830 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:57.203842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:57.204670 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:57.205717 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:57.205750 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.207274 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.207280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:57.209030 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:57.209047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.210726 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.210733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:57.211049 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.211808 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:57.211837 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:57.214633 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:57.218402 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:57.218416 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:57.218441 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:57.219478 (+ 1037us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:57.220113 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:57.220118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:57.220122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:57.220462 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:57.221208 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:57.221215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:57.221603 (+ 388us) client_negotiation.cc:769] Sending connection context
0812 16:44:57.222035 (+ 432us) client_negotiation.cc:240] Negotiation successful
0812 16:44:57.222527 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":344,"threads_started":1}
I20260812 16:44:57.226050 4377 heartbeater.cc:347] Connected to a master server at 127.28.243.62:34161
I20260812 16:44:57.226506 4377 heartbeater.cc:464] Registering TS with master...
I20260812 16:44:57.227398 4377 heartbeater.cc:511] Master 127.28.243.62:34161 requested a full tablet report, sending...
I20260812 16:44:57.229809 3605 ts_manager.cc:194] Registered new tserver with Master: 96297c13a37848869d3195390613edb1 (127.28.243.3:40147)
I20260812 16:44:57.230816 3605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:60925
I20260812 16:44:57.234397 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:44:57.298355 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.279592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.28.243.1:36293)
0812 16:44:57.280351 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:57.280361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:57.280720 (+ 359us) server_negotiation.cc:407] Connection header received
0812 16:44:57.280911 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:57.280919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:57.281135 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:57.281295 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:57.282902 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.283987 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.285270 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.285862 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.285984 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.291463 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:57.291489 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:57.291514 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:57.291572 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:57.295164 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.296080 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.296089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.296092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.296273 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.296670 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.296676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.296678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.297067 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:57.297398 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:57.297700 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 16:44:57.297977 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":357,"threads_started":1}
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:44:57.319629 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.303444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.28.243.62:34161)
0812 16:44:57.303674 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:57.303681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:57.303825 (+ 144us) server_negotiation.cc:407] Connection header received
0812 16:44:57.304031 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:57.304037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:57.304117 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:57.304213 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:57.305533 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.306305 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.307571 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.307991 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.308087 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.313099 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:57.313106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:57.313110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:57.313152 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:57.315984 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.316965 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.316974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.316977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.317269 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.317823 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.317829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.317832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.318282 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:57.318500 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:57.319150 (+ 650us) server_negotiation.cc:299] Negotiation successful
0812 16:44:57.319382 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 16:44:57.320555 3605 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60428
I20260812 16:44:57.321978 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3718
I20260812 16:44:57.339154 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.323268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.28.243.1:36293)
0812 16:44:57.323491 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:57.323497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:57.323786 (+ 289us) server_negotiation.cc:407] Connection header received
0812 16:44:57.323960 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:57.323964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:57.324070 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:57.324159 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:57.325678 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.326382 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.327717 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.328169 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.328289 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.332023 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:57.332031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:57.332035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:57.332089 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:57.335685 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.336661 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.336666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.336669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.336815 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.337224 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.337230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.337233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.337528 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:57.337751 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:57.338458 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 16:44:57.338737 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:44:57.340166 3930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:57.426952 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3718
I20260812 16:44:57.451587 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3951
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:44:57.470330 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.452718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.28.243.2:46707)
0812 16:44:57.453300 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:57.453307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:57.453334 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:44:57.453408 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:57.453414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:57.453623 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:57.453778 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:57.455170 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.456222 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.457774 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.458374 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.458489 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.463645 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:57.463661 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:57.463680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:57.463736 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:57.467195 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.468017 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.468026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.468028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.468176 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.468561 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.468565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.468568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.468977 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:57.469292 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:57.469653 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 16:44:57.469964 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":283,"threads_started":1}
I20260812 16:44:57.471930 4155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:57.564440 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3951
I20260812 16:44:57.596046 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4171
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[3517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:44:57.616072 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:57.597282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.28.243.3:40147)
0812 16:44:57.598015 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:57.598023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:57.598056 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:44:57.598127 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:57.598134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:57.598350 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:57.598491 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:57.599868 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.600957 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.602417 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:57.603096 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:57.603212 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:57.608455 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:57.608484 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:57.608509 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:57.608577 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:57.612627 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.613571 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.613582 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.613584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.613753 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:57.614216 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:57.614222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:57.614224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:57.614649 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:57.614967 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:57.615390 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 16:44:57.615698 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":415,"threads_started":1}
I20260812 16:44:57.617879 4367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:57.711956 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4171
I20260812 16:44:57.742512 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3544
I20260812 16:44:57.743868 3656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:57.851691 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3544
2026-08-12T16:44:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4554 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[4416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[4416](info): set up 2 sockets
Aug 12 16:44:57 dist-test-slave-1vmg krb5kdc[4416](info): commencing operation
krb5kdc: starting...
W20260812 16:44:59.157618 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.007s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:44:59 dist-test-slave-1vmg krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:44:59.460170 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:44937
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:59.821493 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:59.821842 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:59.821991 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:59.829816 4432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:59.829969 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:59.830018 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:59.830119 4432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:59.830199 4432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:59.840988 4432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:44937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:59.842458 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:59.843973 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:59.858594 4441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:59.858649 4442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:59.858673 4444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:59.859745 4432 server_base.cc:1034] running on GCE node
I20260812 16:44:59.860606 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:59.862280 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:59.863631 4432 hybrid_clock.cc:648] HybridClock initialized: now 1786553099863544 us; error 84 us; skew 500 ppm
I20260812 16:44:59.864040 4432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:59 dist-test-slave-1vmg krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:59.867797 4432 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:44:59.869891 4432 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:59.870009 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:59.870450 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:59.874496 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "89fd035125504abdb5a1dbe25d6089f8"
format_stamp: "Formatted at 2026-08-12 16:44:59 on dist-test-slave-1vmg"
server_key: "e1cc0a67efebee49c2ccac7f64a7feff"
server_key_iv: "1e4b1b4a2fa68ae96bcf3c6ced0db9e5"
server_key_version: "encryptionkey@0"
I20260812 16:44:59.875815 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "89fd035125504abdb5a1dbe25d6089f8"
format_stamp: "Formatted at 2026-08-12 16:44:59 on dist-test-slave-1vmg"
server_key: "e1cc0a67efebee49c2ccac7f64a7feff"
server_key_iv: "1e4b1b4a2fa68ae96bcf3c6ced0db9e5"
server_key_version: "encryptionkey@0"
I20260812 16:44:59.883502 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 16:44:59.889238 4454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:59.891490 4432 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:44:59.892163 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "89fd035125504abdb5a1dbe25d6089f8"
format_stamp: "Formatted at 2026-08-12 16:44:59 on dist-test-slave-1vmg"
server_key: "e1cc0a67efebee49c2ccac7f64a7feff"
server_key_iv: "1e4b1b4a2fa68ae96bcf3c6ced0db9e5"
server_key_version: "encryptionkey@0"
I20260812 16:44:59.892447 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:59.944708 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:59.950237 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:59.950565 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:59.972573 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:44937 every 8 connection(s)
I20260812 16:44:59.972637 4432 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:44937
I20260812 16:44:59.974416 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:59.978829 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:59.982120 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4432
I20260812 16:44:59.982362 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:44:59.983130 29644 external_mini_cluster.cc:1428] Setting key cbe6204dc5c1c463e8e686554e8dd4d5
I20260812 16:44:59.991564 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8: Bootstrap starting.
I20260812 16:44:59.995108 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:59.996336 4534 log.cc:824] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:59.999914 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8: No bootstrap required, opened a new log
Aug 12 16:44:59 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:00.008522 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fd035125504abdb5a1dbe25d6089f8" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44937 } }
I20260812 16:45:00.008914 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:00.009012 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fd035125504abdb5a1dbe25d6089f8, State: Initialized, Role: FOLLOWER
I20260812 16:45:00.009620 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fd035125504abdb5a1dbe25d6089f8" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44937 } }
I20260812 16:45:00.009866 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:00.010094 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:00.010350 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:00.013010 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:59.985538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.28.243.62:44937)
0812 16:44:59.986814 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:59.986849 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:59.986909 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:44:59.990365 (+ 3456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:59.990396 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:59.990869 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:59.991514 (+ 645us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:59.993944 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:59.996135 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:59.997929 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:59.998759 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:59.998985 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:00.004079 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:00.004090 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:00.004110 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:00.004164 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:00.007620 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:00.008431 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:00.008438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:00.008445 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:00.008636 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:00.009050 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:00.009054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:00.009056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:00.009759 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:00.010317 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:00.010801 (+ 484us) server_negotiation.cc:299] Negotiation successful
0812 16:45:00.011609 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":530,"threads_started":1}
I20260812 16:45:00.013100 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fd035125504abdb5a1dbe25d6089f8" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44937 } }
I20260812 16:45:00.013742 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fd035125504abdb5a1dbe25d6089f8; no voters:
I20260812 16:45:00.014381 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:00.014518 4543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:00.016871 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:00.017165 4543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [term 1 LEADER]: Becoming Leader. State: Replica: 89fd035125504abdb5a1dbe25d6089f8, State: Running, Role: LEADER
I20260812 16:45:00.018181 4543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fd035125504abdb5a1dbe25d6089f8" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44937 } }
I20260812 16:45:00.021626 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fd035125504abdb5a1dbe25d6089f8. Latest consensus state: current_term: 1 leader_uuid: "89fd035125504abdb5a1dbe25d6089f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fd035125504abdb5a1dbe25d6089f8" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44937 } } }
I20260812 16:45:00.022114 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:00.021622 4546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89fd035125504abdb5a1dbe25d6089f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fd035125504abdb5a1dbe25d6089f8" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44937 } } }
I20260812 16:45:00.022362 4546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:00.023090 4569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:00.028718 4569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:00.040086 4569 catalog_manager.cc:1347] Generated new cluster ID: 025300a7aef54a248011e4ee36334d63
I20260812 16:45:00.040237 4569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:00.061336 4569 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:00.063586 4569 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:00.075606 4569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8: Generated new TSK 0
I20260812 16:45:00.076536 4569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:45:00.157792 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44937
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:00.550575 4581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:00.551010 4581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:00.551206 4581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:00.559834 4581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:00.559990 4581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:00.560149 4581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:00.572834 4581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:00.574841 4581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:00.576655 4581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:00.592567 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:00.592924 4591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:00.592890 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:00.597026 4581 server_base.cc:1034] running on GCE node
I20260812 16:45:00.597813 4581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:00.599143 4581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:00.600445 4581 hybrid_clock.cc:648] HybridClock initialized: now 1786553100600380 us; error 54 us; skew 500 ppm
I20260812 16:45:00.600839 4581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:00 dist-test-slave-1vmg krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:00.608345 4581 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:45:00.609771 4581 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:00.609875 4581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:00.610221 4581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:00.613516 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "903b7c1846374ff18a00a240fd683ed1"
format_stamp: "Formatted at 2026-08-12 16:45:00 on dist-test-slave-1vmg"
server_key: "f226bc6bf5ab3824b48edb6a8e1bd874"
server_key_iv: "abaa615ed53dac6a5ba3de5c13095274"
server_key_version: "encryptionkey@0"
I20260812 16:45:00.614395 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "903b7c1846374ff18a00a240fd683ed1"
format_stamp: "Formatted at 2026-08-12 16:45:00 on dist-test-slave-1vmg"
server_key: "f226bc6bf5ab3824b48edb6a8e1bd874"
server_key_iv: "abaa615ed53dac6a5ba3de5c13095274"
server_key_version: "encryptionkey@0"
I20260812 16:45:00.620209 4581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 16:45:00.624315 4606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:00.626320 4581 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:45:00.626508 4581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "903b7c1846374ff18a00a240fd683ed1"
format_stamp: "Formatted at 2026-08-12 16:45:00 on dist-test-slave-1vmg"
server_key: "f226bc6bf5ab3824b48edb6a8e1bd874"
server_key_iv: "abaa615ed53dac6a5ba3de5c13095274"
server_key_version: "encryptionkey@0"
I20260812 16:45:00.626744 4581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:00.639266 4581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:00.643859 4581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:00.644115 4581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:00.645123 4581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:00.646512 4581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:00.646617 4581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:00.646733 4581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:00.646804 4581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:00.682539 4581 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:38761
I20260812 16:45:00.682596 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:38761 every 8 connection(s)
I20260812 16:45:00.683485 4581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:00.688501 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4581
I20260812 16:45:00.688803 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:45:00.690128 29644 external_mini_cluster.cc:1428] Setting key d80c9641df81120e9ea4f140a431f25e
Aug 12 16:45:00 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:00.712124 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:00.686833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:53107 (local address 127.28.243.62:44937)
0812 16:45:00.687580 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:00.687586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:00.688731 (+ 1145us) server_negotiation.cc:407] Connection header received
0812 16:45:00.692144 (+ 3413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:00.692149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:00.692237 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:00.692357 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:00.695257 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:00.696007 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:00.698546 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:00.699184 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:00.699429 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:00.703834 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:00.703841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:00.703846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:00.703893 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:00.706728 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:00.707873 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:00.707878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:00.707880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:00.708036 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:00.709495 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:00.709500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:00.709503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:00.709820 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:00.709955 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:00.711472 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0812 16:45:00.711786 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":302,"threads_started":1}
I20260812 16:45:00.714375 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:00.687318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44937 (local address 127.28.243.1:53107)
0812 16:45:00.688423 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:00.688515 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:00.691601 (+ 3086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:00.692742 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:00.692754 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:00.693716 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:00.694953 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:00.694995 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:00.696205 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:00.696210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:00.698222 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:00.698242 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:00.699499 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:00.699506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:00.699824 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:00.700747 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:00.700779 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:00.703509 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:00.706983 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:00.706994 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:00.707016 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:00.707663 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:00.708211 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:00.708224 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:00.708227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:00.709186 (+ 959us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:00.709991 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:00.710001 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:00.710564 (+ 563us) client_negotiation.cc:769] Sending connection context
0812 16:45:00.711227 (+ 663us) client_negotiation.cc:240] Negotiation successful
0812 16:45:00.711866 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":401,"threads_started":1}
I20260812 16:45:00.716207 4799 heartbeater.cc:347] Connected to a master server at 127.28.243.62:44937
I20260812 16:45:00.716529 4799 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:00.717293 4799 heartbeater.cc:511] Master 127.28.243.62:44937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:45:00.720386 4480 ts_manager.cc:194] Registered new tserver with Master: 903b7c1846374ff18a00a240fd683ed1 (127.28.243.1:38761)
I20260812 16:45:00.721980 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:53107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:45:00.760466 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44937
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:01.135579 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:01.135882 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:01.136022 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:01.143442 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:01.143577 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:01.143738 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:01.154059 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:01.155591 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:01.157037 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:01.173884 4821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:01.174064 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:01.173861 4822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:01.174583 4812 server_base.cc:1034] running on GCE node
I20260812 16:45:01.175339 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:01.176399 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:01.177625 4812 hybrid_clock.cc:648] HybridClock initialized: now 1786553101177562 us; error 54 us; skew 500 ppm
I20260812 16:45:01.177979 4812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:01 dist-test-slave-1vmg krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553101, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:01.181329 4812 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:45:01.182595 4812 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:01.182689 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:01.183075 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:01.186179 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "99cd262cf2f745f5b2ea6bf0efacbd96"
format_stamp: "Formatted at 2026-08-12 16:45:01 on dist-test-slave-1vmg"
server_key: "a881d1d297032ec782e70da8f60ea6cc"
server_key_iv: "5cbaf247f8a326bed8fc475001dfbf52"
server_key_version: "encryptionkey@0"
I20260812 16:45:01.187074 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "99cd262cf2f745f5b2ea6bf0efacbd96"
format_stamp: "Formatted at 2026-08-12 16:45:01 on dist-test-slave-1vmg"
server_key: "a881d1d297032ec782e70da8f60ea6cc"
server_key_iv: "5cbaf247f8a326bed8fc475001dfbf52"
server_key_version: "encryptionkey@0"
I20260812 16:45:01.192361 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260812 16:45:01.195932 4839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:01.197296 4812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:45:01.197455 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "99cd262cf2f745f5b2ea6bf0efacbd96"
format_stamp: "Formatted at 2026-08-12 16:45:01 on dist-test-slave-1vmg"
server_key: "a881d1d297032ec782e70da8f60ea6cc"
server_key_iv: "5cbaf247f8a326bed8fc475001dfbf52"
server_key_version: "encryptionkey@0"
I20260812 16:45:01.197721 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:01.215577 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:01.221125 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:01.221503 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:01.229863 4812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:01.233791 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:01.233959 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:01.234092 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:01.234185 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:01.270982 4812 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:43567
I20260812 16:45:01.271196 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:43567 every 8 connection(s)
I20260812 16:45:01.271934 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:01.281222 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4812
I20260812 16:45:01.281544 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:45:01.282105 29644 external_mini_cluster.cc:1428] Setting key 82abfbf8bd2904eda8cd2782dc248ce6
Aug 12 16:45:01 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553101, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:01.299317 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.275930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:33121 (local address 127.28.243.62:44937)
0812 16:45:01.276471 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:01.276476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:01.277752 (+ 1276us) server_negotiation.cc:407] Connection header received
0812 16:45:01.279968 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:01.279975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:01.280071 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:01.280171 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:01.281975 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.282749 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.284964 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.285674 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.285795 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.291570 (+ 5775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:01.291579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:01.291584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:01.291643 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:01.295483 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:01.296577 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:01.296582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:01.296587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:01.296765 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:01.297278 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:01.297283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:01.297285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:01.297653 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:01.297806 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:01.298605 (+ 799us) server_negotiation.cc:299] Negotiation successful
0812 16:45:01.298868 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":278,"threads_started":1}
I20260812 16:45:01.300249 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.276672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44937 (local address 127.28.243.2:33121)
0812 16:45:01.277531 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:01.277589 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:01.279619 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:01.280478 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:01.280489 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:01.281086 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:01.281776 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.281802 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.283003 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.283009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.284685 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.284699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.285963 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.285971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.286276 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.287398 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:01.287432 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:01.291107 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:01.295727 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:01.295741 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:01.295764 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:01.296376 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:01.296919 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:01.296923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:01.296926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:01.297136 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:01.297783 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:01.297790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:01.298206 (+ 416us) client_negotiation.cc:769] Sending connection context
0812 16:45:01.298460 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 16:45:01.299025 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":347,"threads_started":1}
I20260812 16:45:01.301885 5021 heartbeater.cc:347] Connected to a master server at 127.28.243.62:44937
I20260812 16:45:01.302228 5021 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:01.303007 5021 heartbeater.cc:511] Master 127.28.243.62:44937 requested a full tablet report, sending...
I20260812 16:45:01.304780 4480 ts_manager.cc:194] Registered new tserver with Master: 99cd262cf2f745f5b2ea6bf0efacbd96 (127.28.243.2:43567)
I20260812 16:45:01.305958 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:33121
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:45:01.349051 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44937
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:01.711267 5031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:01.711608 5031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:01.711720 5031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:01.719875 5031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:01.720311 5031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:01.720474 5031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:01.725492 4799 heartbeater.cc:503] Master 127.28.243.62:44937 was elected leader, sending a full tablet report...
I20260812 16:45:01.732347 5031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:01.733780 5031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:01.735229 5031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:01.750550 5044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:01.750590 5037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:01.750550 5039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:01.751493 5031 server_base.cc:1034] running on GCE node
I20260812 16:45:01.752264 5031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:01.753351 5031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:01.754618 5031 hybrid_clock.cc:648] HybridClock initialized: now 1786553101754570 us; error 56 us; skew 500 ppm
I20260812 16:45:01.755031 5031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:01 dist-test-slave-1vmg krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553101, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:01.759245 5031 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:45:01.761101 5031 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:01.761256 5031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:01.761595 5031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:01.765002 5031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "6a475cf762a54112b7d2d5345f1c6523"
format_stamp: "Formatted at 2026-08-12 16:45:01 on dist-test-slave-1vmg"
server_key: "88061976ab17a2c5df5e2aa2ac0e6030"
server_key_iv: "a02a33874f04b59249e6ebe5a85cfac9"
server_key_version: "encryptionkey@0"
I20260812 16:45:01.765879 5031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "6a475cf762a54112b7d2d5345f1c6523"
format_stamp: "Formatted at 2026-08-12 16:45:01 on dist-test-slave-1vmg"
server_key: "88061976ab17a2c5df5e2aa2ac0e6030"
server_key_iv: "a02a33874f04b59249e6ebe5a85cfac9"
server_key_version: "encryptionkey@0"
I20260812 16:45:01.772387 5031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:45:01.776866 5053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:01.778376 5031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:01.778581 5031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "6a475cf762a54112b7d2d5345f1c6523"
format_stamp: "Formatted at 2026-08-12 16:45:01 on dist-test-slave-1vmg"
server_key: "88061976ab17a2c5df5e2aa2ac0e6030"
server_key_iv: "a02a33874f04b59249e6ebe5a85cfac9"
server_key_version: "encryptionkey@0"
I20260812 16:45:01.778827 5031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:01.801260 5031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:01.807708 5031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:01.808117 5031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:01.809479 5031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:01.811545 5031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:01.811667 5031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:01.811967 5031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:01.812109 5031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:01.853041 5244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:39291 every 8 connection(s)
I20260812 16:45:01.853158 5031 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:39291
I20260812 16:45:01.854823 5031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:01.857055 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 5031
I20260812 16:45:01.857277 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:45:01.857846 29644 external_mini_cluster.cc:1428] Setting key a22c335c813d88eff574008886244a1a
Aug 12 16:45:01 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553101, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:01.879796 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.857920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:58473 (local address 127.28.243.62:44937)
0812 16:45:01.860661 (+ 2741us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:01.860666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:01.860702 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:45:01.862255 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:01.862260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:01.862354 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:01.862455 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:01.864337 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.865139 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.867094 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.867707 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.868259 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.872966 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:01.872973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:01.872977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:01.873029 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:01.876273 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:01.877237 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:01.877241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:01.877243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:01.877381 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:01.877834 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:01.877838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:01.877841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:01.878135 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:01.878288 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:01.879103 (+ 815us) server_negotiation.cc:299] Negotiation successful
0812 16:45:01.879385 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2625,"thread_start_us":2494,"threads_started":1}
I20260812 16:45:01.880936 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.858663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44937 (local address 127.28.243.3:58473)
0812 16:45:01.859855 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:01.859930 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:01.861886 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:01.862743 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:01.862753 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:01.863435 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:01.864103 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.864129 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.865374 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.865379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.866743 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.866759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.868311 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.868317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.868554 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.869517 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:01.869545 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:01.872595 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:01.876484 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:01.876495 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:01.876513 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:01.877077 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:01.877496 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:01.877502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:01.877505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:01.877702 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:01.878275 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:01.878284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:01.878668 (+ 384us) client_negotiation.cc:769] Sending connection context
0812 16:45:01.878980 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 16:45:01.879576 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":452,"threads_started":1}
I20260812 16:45:01.882536 5246 heartbeater.cc:347] Connected to a master server at 127.28.243.62:44937
I20260812 16:45:01.882890 5246 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:01.883730 5246 heartbeater.cc:511] Master 127.28.243.62:44937 requested a full tablet report, sending...
I20260812 16:45:01.885885 4480 ts_manager.cc:194] Registered new tserver with Master: 6a475cf762a54112b7d2d5345f1c6523 (127.28.243.3:39291)
I20260812 16:45:01.887030 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:58473
I20260812 16:45:01.888334 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:45:01.910843 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.894698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.28.243.62:44937)
0812 16:45:01.895039 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:01.895046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:01.895273 (+ 227us) server_negotiation.cc:407] Connection header received
0812 16:45:01.895483 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:01.895488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:01.895561 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:01.895650 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:01.897215 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.898061 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.899506 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.899932 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.900055 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.902600 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:01.902606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:01.902609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:01.902650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:01.905981 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:01.907256 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:01.907261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:01.907263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:01.907468 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:01.908274 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:01.908280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:01.908283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:01.908743 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:01.909295 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:01.910198 (+ 903us) server_negotiation.cc:299] Negotiation successful
0812 16:45:01.910501 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 16:45:01.920292 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:01.928853 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:01.958846 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.946053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:43567 (local address 127.0.0.1:43290)
0812 16:45:01.946980 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:01.947021 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:01.947216 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:01.950210 (+ 2994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:01.950218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:01.950265 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:01.950736 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.950751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.952597 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.952604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.955764 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.955783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.957743 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.957750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.957974 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.958015 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 16:45:01.958148 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 16:45:01.958376 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":454,"threads_started":1}
I20260812 16:45:01.959231 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.946469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.28.243.2:43567)
0812 16:45:01.949125 (+ 2656us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:01.949137 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:01.949177 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:45:01.949305 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:01.949315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:01.949642 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:01.949895 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:01.951026 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.952392 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.955983 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.957491 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.957647 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.957874 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:01.958217 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 16:45:01.958407 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":1314,"threads_started":1}
I20260812 16:45:01.960822 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.946871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:39291 (local address 127.0.0.1:46450)
0812 16:45:01.949240 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:01.949270 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:01.949507 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:01.951688 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:01.951694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:01.951731 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:01.952440 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.952456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.954666 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.954675 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.956761 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.956781 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.958802 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.958808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.960178 (+ 1370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.960217 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:45:01.960316 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 16:45:01.960464 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":1906,"threads_started":1}
I20260812 16:45:01.960916 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.949734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.28.243.3:39291)
0812 16:45:01.950682 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:01.950691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:01.950738 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:45:01.950893 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:01.950904 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:01.951355 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:01.951681 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:01.952727 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.954460 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.956978 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.958547 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.959714 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.959993 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:01.960382 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 16:45:01.960566 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":575,"threads_started":1}
I20260812 16:45:01.962582 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.948187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:38761 (local address 127.0.0.1:37696)
0812 16:45:01.952568 (+ 4381us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:01.952590 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:01.952763 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:01.953747 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:01.953753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:01.953794 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:01.954607 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.954623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.956479 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.956483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.958529 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:01.958543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.960764 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.960770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:01.960943 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.960976 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:45:01.962093 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0812 16:45:01.962246 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4206,"thread_start_us":3955,"threads_started":1}
I20260812 16:45:01.962752 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:01.949157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.28.243.1:38761)
0812 16:45:01.950126 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:01.950137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:01.952757 (+ 2620us) server_negotiation.cc:407] Connection header received
0812 16:45:01.952973 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:01.952981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:01.953318 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:01.953584 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:01.954860 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.956329 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.958758 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:01.960232 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:01.960379 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:01.960601 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:01.962185 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0812 16:45:01.962386 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":941,"threads_started":1}
I20260812 16:45:01.963752 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 53e4429040e24641b3bcb9a3e9e4046d (DEFAULT_TABLE table=test-table [id=56ddaa75e627445c918f24c672effce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:01.964901 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 53e4429040e24641b3bcb9a3e9e4046d (DEFAULT_TABLE table=test-table [id=56ddaa75e627445c918f24c672effce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:01.966079 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e4429040e24641b3bcb9a3e9e4046d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:01.967202 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e4429040e24641b3bcb9a3e9e4046d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:01.967299 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 53e4429040e24641b3bcb9a3e9e4046d (DEFAULT_TABLE table=test-table [id=56ddaa75e627445c918f24c672effce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:01.968881 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e4429040e24641b3bcb9a3e9e4046d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:01.996064 5276 tablet_bootstrap.cc:492] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Bootstrap starting.
I20260812 16:45:01.998147 5283 tablet_bootstrap.cc:492] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Bootstrap starting.
I20260812 16:45:01.998618 5285 tablet_bootstrap.cc:492] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Bootstrap starting.
I20260812 16:45:02.001665 5283 tablet_bootstrap.cc:654] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:02.002858 5285 tablet_bootstrap.cc:654] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:02.002976 5283 log.cc:824] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:02.006755 5283 tablet_bootstrap.cc:492] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: No bootstrap required, opened a new log
I20260812 16:45:02.007257 5283 ts_tablet_manager.cc:1397] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260812 16:45:02.007696 5285 log.cc:824] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:02.008610 5276 tablet_bootstrap.cc:654] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:02.010154 5276 log.cc:824] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:02.015203 5285 tablet_bootstrap.cc:492] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: No bootstrap required, opened a new log
I20260812 16:45:02.015592 5285 ts_tablet_manager.cc:1397] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260812 16:45:02.017655 5276 tablet_bootstrap.cc:492] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: No bootstrap required, opened a new log
I20260812 16:45:02.018123 5276 ts_tablet_manager.cc:1397] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20260812 16:45:02.018174 5283 raft_consensus.cc:348] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.018838 5283 raft_consensus.cc:374] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:02.019366 5283 raft_consensus.cc:729] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903b7c1846374ff18a00a240fd683ed1, State: Initialized, Role: FOLLOWER
I20260812 16:45:02.020260 5283 consensus_queue.cc:260] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.026439 5283 ts_tablet_manager.cc:1428] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260812 16:45:02.026561 5285 raft_consensus.cc:348] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.027330 5285 raft_consensus.cc:374] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:02.027454 5285 raft_consensus.cc:729] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cd262cf2f745f5b2ea6bf0efacbd96, State: Initialized, Role: FOLLOWER
I20260812 16:45:02.028663 5285 consensus_queue.cc:260] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.029978 5276 raft_consensus.cc:348] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.030424 5276 raft_consensus.cc:374] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:02.030501 5276 raft_consensus.cc:729] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a475cf762a54112b7d2d5345f1c6523, State: Initialized, Role: FOLLOWER
I20260812 16:45:02.031160 5276 consensus_queue.cc:260] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.032876 5246 heartbeater.cc:503] Master 127.28.243.62:44937 was elected leader, sending a full tablet report...
I20260812 16:45:02.032943 5276 ts_tablet_manager.cc:1428] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260812 16:45:02.034858 5021 heartbeater.cc:503] Master 127.28.243.62:44937 was elected leader, sending a full tablet report...
I20260812 16:45:02.035746 5285 ts_tablet_manager.cc:1428] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260812 16:45:02.068099 5290 raft_consensus.cc:482] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:02.068410 5290 raft_consensus.cc:504] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.070387 5290 leader_election.cc:288] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99cd262cf2f745f5b2ea6bf0efacbd96 (127.28.243.2:43567), 6a475cf762a54112b7d2d5345f1c6523 (127.28.243.3:39291)
I20260812 16:45:02.080842 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.071188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:57783 (local address 127.28.243.3:39291)
0812 16:45:02.071391 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:02.071396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:02.072275 (+ 879us) server_negotiation.cc:407] Connection header received
0812 16:45:02.072518 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:02.072524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:02.072607 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:02.072715 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:02.073634 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.074759 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.078046 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.079390 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.079597 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.079682 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:02.080405 (+ 723us) server_negotiation.cc:299] Negotiation successful
0812 16:45:02.080564 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:45:02.081071 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.070697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:43567 (local address 127.28.243.1:46987)
0812 16:45:02.072086 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:02.072112 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:02.072302 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:02.072778 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:02.072783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:02.072836 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:02.073420 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:02.073434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.074977 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.074983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:02.077663 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:02.077684 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.079822 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.079832 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:02.080106 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.080209 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 16:45:02.080318 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 16:45:02.080555 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"mutex_wait_us":266,"spinlock_wait_cycles":56576,"thread_start_us":1088,"threads_started":1}
I20260812 16:45:02.081298 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.071091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:46987 (local address 127.28.243.2:43567)
0812 16:45:02.071336 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:02.071342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:02.072275 (+ 933us) server_negotiation.cc:407] Connection header received
0812 16:45:02.072519 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:02.072524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:02.072607 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:02.072715 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:02.073634 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.074758 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.078058 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.079390 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.079606 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.079682 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:02.080405 (+ 723us) server_negotiation.cc:299] Negotiation successful
0812 16:45:02.081048 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 16:45:02.081511 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.070827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:39291 (local address 127.28.243.1:57783)
0812 16:45:02.072111 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:02.072126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:02.072296 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:02.072778 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:02.072783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:02.072835 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:02.073424 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:02.073434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.074977 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.074983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:02.077658 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:02.077684 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.079843 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.079849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:02.080109 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.080200 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 16:45:02.080318 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 16:45:02.080555 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":227,"spinlock_wait_cycles":22144,"thread_start_us":806,"threads_started":1}
I20260812 16:45:02.082638 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e4429040e24641b3bcb9a3e9e4046d" candidate_uuid: "903b7c1846374ff18a00a240fd683ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a475cf762a54112b7d2d5345f1c6523" is_pre_election: true
I20260812 16:45:02.082679 4942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e4429040e24641b3bcb9a3e9e4046d" candidate_uuid: "903b7c1846374ff18a00a240fd683ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" is_pre_election: true
I20260812 16:45:02.083318 5173 raft_consensus.cc:2454] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903b7c1846374ff18a00a240fd683ed1 in term 0.
I20260812 16:45:02.083454 4942 raft_consensus.cc:2454] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903b7c1846374ff18a00a240fd683ed1 in term 0.
I20260812 16:45:02.084053 4607 leader_election.cc:302] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a475cf762a54112b7d2d5345f1c6523, 903b7c1846374ff18a00a240fd683ed1; no voters:
I20260812 16:45:02.084492 5290 raft_consensus.cc:2781] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:02.084642 5290 raft_consensus.cc:482] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:02.084812 5290 raft_consensus.cc:3037] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:02.087212 5290 raft_consensus.cc:504] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.087926 5290 leader_election.cc:288] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [CANDIDATE]: Term 1 election: Requested vote from peers 99cd262cf2f745f5b2ea6bf0efacbd96 (127.28.243.2:43567), 6a475cf762a54112b7d2d5345f1c6523 (127.28.243.3:39291)
I20260812 16:45:02.089082 4942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e4429040e24641b3bcb9a3e9e4046d" candidate_uuid: "903b7c1846374ff18a00a240fd683ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96"
I20260812 16:45:02.089222 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e4429040e24641b3bcb9a3e9e4046d" candidate_uuid: "903b7c1846374ff18a00a240fd683ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a475cf762a54112b7d2d5345f1c6523"
I20260812 16:45:02.089345 4942 raft_consensus.cc:3037] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:02.089561 5173 raft_consensus.cc:3037] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:02.092327 4942 raft_consensus.cc:2454] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903b7c1846374ff18a00a240fd683ed1 in term 1.
I20260812 16:45:02.092952 4610 leader_election.cc:302] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903b7c1846374ff18a00a240fd683ed1, 99cd262cf2f745f5b2ea6bf0efacbd96; no voters:
I20260812 16:45:02.093279 5290 raft_consensus.cc:2781] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:02.093851 5290 raft_consensus.cc:686] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 1 LEADER]: Becoming Leader. State: Replica: 903b7c1846374ff18a00a240fd683ed1, State: Running, Role: LEADER
I20260812 16:45:02.094596 5290 consensus_queue.cc:237] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } }
I20260812 16:45:02.094766 5173 raft_consensus.cc:2454] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903b7c1846374ff18a00a240fd683ed1 in term 1.
I20260812 16:45:02.100970 4480 catalog_manager.cc:5495] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 903b7c1846374ff18a00a240fd683ed1 (127.28.243.1). New cstate: current_term: 1 leader_uuid: "903b7c1846374ff18a00a240fd683ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903b7c1846374ff18a00a240fd683ed1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 38761 } health_report { overall_health: HEALTHY } } }
W20260812 16:45:02.107254 5247 tablet.cc:2369] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:02.141212 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.133471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.28.243.1:38761)
0812 16:45:02.133782 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:02.133790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:02.133823 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:45:02.133961 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:02.133966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:02.134083 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:02.134304 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:02.135103 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.136487 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.138150 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.138839 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.139049 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.139676 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:02.140475 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:02.140656 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:02.140794 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 16:45:02.140898 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 16:45:02.156042 4942 raft_consensus.cc:1261] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 1 FOLLOWER]: Refusing update from remote peer 903b7c1846374ff18a00a240fd683ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:02.156397 5173 raft_consensus.cc:1261] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 1 FOLLOWER]: Refusing update from remote peer 903b7c1846374ff18a00a240fd683ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:02.157306 5290 consensus_queue.cc:1035] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99cd262cf2f745f5b2ea6bf0efacbd96" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:02.157697 5299 consensus_queue.cc:1035] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a475cf762a54112b7d2d5345f1c6523" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:02.177735 5306 mvcc.cc:205] Tried to move back new op lower bound from 7317721506411425792 to 7317721506184073216. Current Snapshot: MvccSnapshot[applied={T|T < 7317721506411425792}]
I20260812 16:45:02.178515 5309 mvcc.cc:205] Tried to move back new op lower bound from 7317721506411425792 to 7317721506184073216. Current Snapshot: MvccSnapshot[applied={T|T < 7317721506411425792}]
I20260812 16:45:02.194366 4480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:02.194624 4480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:02.198629 4480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89fd035125504abdb5a1dbe25d6089f8: Sending DeleteTablet for 3 replicas of tablet 53e4429040e24641b3bcb9a3e9e4046d
I20260812 16:45:02.200034 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e4429040e24641b3bcb9a3e9e4046d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:43290
I20260812 16:45:02.200152 4680 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e4429040e24641b3bcb9a3e9e4046d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:37696
I20260812 16:45:02.200525 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e4429040e24641b3bcb9a3e9e4046d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:46450
I20260812 16:45:02.203500 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4581
I20260812 16:45:02.209906 5335 tablet_replica.cc:330] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: stopping tablet replica
I20260812 16:45:02.210503 5335 raft_consensus.cc:2229] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:02.211158 5335 raft_consensus.cc:2258] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:02.216581 5335 ts_tablet_manager.cc:1905] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:02.218321 5334 tablet_replica.cc:330] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: stopping tablet replica
I20260812 16:45:02.218876 5334 raft_consensus.cc:2229] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:02.219944 5334 raft_consensus.cc:2258] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:02.223418 5334 ts_tablet_manager.cc:1905] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:02.236063 5332 tablet_replica.cc:330] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: stopping tablet replica
I20260812 16:45:02.236480 5335 ts_tablet_manager.cc:1918] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:02.236706 5335 log.cc:1196] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/53e4429040e24641b3bcb9a3e9e4046d
I20260812 16:45:02.237270 5335 ts_tablet_manager.cc:1939] T 53e4429040e24641b3bcb9a3e9e4046d P 6a475cf762a54112b7d2d5345f1c6523: Deleting consensus metadata
Aug 12 16:45:02 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:45:02.239647 4455 catalog_manager.cc:4841] TS 6a475cf762a54112b7d2d5345f1c6523 (127.28.243.3:39291): tablet 53e4429040e24641b3bcb9a3e9e4046d (table test-table [id=56ddaa75e627445c918f24c672effce8]) successfully deleted
I20260812 16:45:02.240383 5332 raft_consensus.cc:2229] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:02.240922 5332 raft_consensus.cc:2258] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:02.243403 5332 ts_tablet_manager.cc:1905] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:02.246254 5334 ts_tablet_manager.cc:1918] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:02.246480 5334 log.cc:1196] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/53e4429040e24641b3bcb9a3e9e4046d
I20260812 16:45:02.247114 5334 ts_tablet_manager.cc:1939] T 53e4429040e24641b3bcb9a3e9e4046d P 903b7c1846374ff18a00a240fd683ed1: Deleting consensus metadata
I20260812 16:45:02.249667 4458 catalog_manager.cc:4841] TS 903b7c1846374ff18a00a240fd683ed1 (127.28.243.1:38761): tablet 53e4429040e24641b3bcb9a3e9e4046d (table test-table [id=56ddaa75e627445c918f24c672effce8]) successfully deleted
I20260812 16:45:02.253856 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.205616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.28.243.1:38761)
0812 16:45:02.205839 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:02.205846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:02.217542 (+ 11696us) server_negotiation.cc:407] Connection header received
0812 16:45:02.218168 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:02.218175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:02.218292 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:02.218383 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:02.224079 (+ 5696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.225644 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.231034 (+ 5390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.231894 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.232029 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.244620 (+ 12591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:02.244639 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:02.244660 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:02.244729 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:02.249724 (+ 4995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:02.250761 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:02.250772 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:02.250775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:02.251038 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:02.251556 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:02.251561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:02.251565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:02.252111 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:02.252419 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:02.253114 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 16:45:02.253432 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:45:02.254655 5332 ts_tablet_manager.cc:1918] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:02.254868 5332 log.cc:1196] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/53e4429040e24641b3bcb9a3e9e4046d
I20260812 16:45:02.255558 5332 ts_tablet_manager.cc:1939] T 53e4429040e24641b3bcb9a3e9e4046d P 99cd262cf2f745f5b2ea6bf0efacbd96: Deleting consensus metadata
I20260812 16:45:02.255645 4786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:02.257768 4461 catalog_manager.cc:4841] TS 99cd262cf2f745f5b2ea6bf0efacbd96 (127.28.243.2:43567): tablet 53e4429040e24641b3bcb9a3e9e4046d (table test-table [id=56ddaa75e627445c918f24c672effce8]) successfully deleted
I20260812 16:45:02.383831 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4581
I20260812 16:45:02.415292 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4812
Aug 12 16:45:02 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:45:02.435053 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.416083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.28.243.2:43567)
0812 16:45:02.416278 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:02.416282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:02.416302 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:45:02.416402 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:02.416406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:02.416491 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:02.416598 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:02.417889 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.418679 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.420187 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.420667 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.420811 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.426571 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:02.426599 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:02.426624 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:02.426693 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:02.430693 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:02.431916 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:02.431928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:02.431930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:02.432151 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:02.432679 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:02.432683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:02.432686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:02.433361 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:02.433675 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:02.434440 (+ 765us) server_negotiation.cc:299] Negotiation successful
0812 16:45:02.434687 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:45:02.435706 5014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:02.529585 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4812
I20260812 16:45:02.557420 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 5031
Aug 12 16:45:02 dist-test-slave-1vmg krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:45:02.573874 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:02.558229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.28.243.3:39291)
0812 16:45:02.558441 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:02.558447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:02.558469 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:45:02.558537 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:02.558542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:02.558611 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:02.558718 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:02.560274 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.561009 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.562315 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:02.562721 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:02.562843 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:02.568198 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:02.568227 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:02.568254 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:02.568418 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:02.571372 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:02.572083 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:02.572090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:02.572093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:02.572266 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:02.572590 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:02.572594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:02.572596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:02.572981 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:02.573199 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:02.573417 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 16:45:02.573595 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:45:02.574666 5239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:02.673002 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 5031
I20260812 16:45:02.701747 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4432
I20260812 16:45:02.703037 4527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:02.815562 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4432
2026-08-12T16:45:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4963 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T16:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:02Z Disabled control of system clock
I20260812 16:45:02.919576 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43407
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 16:45:03.340476 5349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:03.340842 5349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:03.340976 5349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:03.348965 5349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:03.349120 5349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:03.349192 5349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:03.349292 5349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:03.349354 5349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:03.361320 5349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:03.362802 5349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:03.364809 5349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:03.380537 5357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:03.380584 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:03.380561 5358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:03.382114 5349 server_base.cc:1034] running on GCE node
I20260812 16:45:03.383111 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:03.384778 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:03.386245 5349 hybrid_clock.cc:648] HybridClock initialized: now 1786553103386159 us; error 70 us; skew 500 ppm
I20260812 16:45:03.386685 5349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:03.389842 5349 webserver.cc:460] Webserver started at http://127.28.243.62:40435/ using document root <none> and password file <none>
I20260812 16:45:03.390688 5349 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:03.390807 5349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:03.391197 5349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:03.394496 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990"
format_stamp: "Formatted at 2026-08-12 16:45:03 on dist-test-slave-1vmg"
server_key: "c2c2afc4c9129c6bfb38880a2664c882"
server_key_iv: "a9de02b1e44d54fb781eb73325a94bdc"
server_key_version: "encryptionkey@0"
I20260812 16:45:03.395427 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990"
format_stamp: "Formatted at 2026-08-12 16:45:03 on dist-test-slave-1vmg"
server_key: "c2c2afc4c9129c6bfb38880a2664c882"
server_key_iv: "a9de02b1e44d54fb781eb73325a94bdc"
server_key_version: "encryptionkey@0"
I20260812 16:45:03.401374 5349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 16:45:03.406250 5369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:03.408540 5349 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 16:45:03.408883 5349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990"
format_stamp: "Formatted at 2026-08-12 16:45:03 on dist-test-slave-1vmg"
server_key: "c2c2afc4c9129c6bfb38880a2664c882"
server_key_iv: "a9de02b1e44d54fb781eb73325a94bdc"
server_key_version: "encryptionkey@0"
I20260812 16:45:03.409238 5349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:03.416400 5349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:03.417369 5349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:03.417693 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:03.439733 5349 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:43407
I20260812 16:45:03.439754 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:43407 every 8 connection(s)
I20260812 16:45:03.441284 5349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:03.445544 5459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:03.451475 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 5349
I20260812 16:45:03.451653 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:03.452721 29644 external_mini_cluster.cc:1428] Setting key e8e885eee338b641d112a2200c4ee2a8
I20260812 16:45:03.459967 5459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990: Bootstrap starting.
I20260812 16:45:03.462743 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:03.455583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.28.243.62:43407)
0812 16:45:03.456908 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:03.456919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:03.456971 (+ 52us) server_negotiation.cc:407] Connection header received
0812 16:45:03.458801 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:03.458828 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:03.459356 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:03.459724 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:03.461023 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:03.461030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:03.461045 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:03.461068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:03.461132 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:03.461148 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:03.461332 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:03.461436 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 16:45:03.461696 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":424,"threads_started":1}
I20260812 16:45:03.463871 5459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:03.464998 5459 log.cc:824] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:03.468168 5459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990: No bootstrap required, opened a new log
I20260812 16:45:03.475776 5459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43407 } }
I20260812 16:45:03.476218 5459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:03.476318 5459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0f65a5cb2c4ad7a5e747faf2c73990, State: Initialized, Role: FOLLOWER
I20260812 16:45:03.476972 5459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43407 } }
I20260812 16:45:03.477214 5459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:03.477348 5459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:03.477514 5459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:03.479843 5459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43407 } }
I20260812 16:45:03.480394 5459 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0f65a5cb2c4ad7a5e747faf2c73990; no voters:
I20260812 16:45:03.481091 5459 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:03.481266 5467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:03.481959 5467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0f65a5cb2c4ad7a5e747faf2c73990, State: Running, Role: LEADER
I20260812 16:45:03.482537 5467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43407 } }
I20260812 16:45:03.483381 5459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:03.486295 5471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0f65a5cb2c4ad7a5e747faf2c73990. Latest consensus state: current_term: 1 leader_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43407 } } }
I20260812 16:45:03.486505 5470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f65a5cb2c4ad7a5e747faf2c73990" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43407 } } }
I20260812 16:45:03.487382 5471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:03.487644 5470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:03.495136 5483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:03.501113 5483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:03.513415 5483 catalog_manager.cc:1347] Generated new cluster ID: c8e361b1100c49d6b63597d672b312f0
I20260812 16:45:03.513607 5483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:03.543244 5483 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:03.543843 5483 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:03.552997 5483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990: Generated new TSK 0
I20260812 16:45:03.554895 5483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:45:03.566331 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43407
--builtin_ntp_servers=127.28.243.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 16:45:03.973435 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:03.973788 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:03.973884 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:03.982832 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:03.983031 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:03.983167 5504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:03.993543 5504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:03.995059 5504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:03.997222 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:04.013386 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:04.013440 5520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:04.013465 5518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:04.014451 5504 server_base.cc:1034] running on GCE node
I20260812 16:45:04.015316 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:04.016644 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:04.017892 5504 hybrid_clock.cc:648] HybridClock initialized: now 1786553104017819 us; error 73 us; skew 500 ppm
I20260812 16:45:04.018271 5504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:04.021692 5504 webserver.cc:460] Webserver started at http://127.28.243.1:36715/ using document root <none> and password file <none>
I20260812 16:45:04.022537 5504 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:04.022665 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:04.023080 5504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:04.028067 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "f8adb6c6016040de9a2491dc8d2c7e53"
format_stamp: "Formatted at 2026-08-12 16:45:04 on dist-test-slave-1vmg"
server_key: "216ef4e2312ff260030c35a0caa0816f"
server_key_iv: "4359dfd3c8d2061331c4800f710edfd7"
server_key_version: "encryptionkey@0"
I20260812 16:45:04.029460 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "f8adb6c6016040de9a2491dc8d2c7e53"
format_stamp: "Formatted at 2026-08-12 16:45:04 on dist-test-slave-1vmg"
server_key: "216ef4e2312ff260030c35a0caa0816f"
server_key_iv: "4359dfd3c8d2061331c4800f710edfd7"
server_key_version: "encryptionkey@0"
I20260812 16:45:04.038007 5504 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20260812 16:45:04.042688 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:04.044482 5504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:04.044926 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "f8adb6c6016040de9a2491dc8d2c7e53"
format_stamp: "Formatted at 2026-08-12 16:45:04 on dist-test-slave-1vmg"
server_key: "216ef4e2312ff260030c35a0caa0816f"
server_key_iv: "4359dfd3c8d2061331c4800f710edfd7"
server_key_version: "encryptionkey@0"
I20260812 16:45:04.045365 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:04.054343 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:04.055747 5504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:04.056262 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:04.057551 5504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:04.059340 5504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:04.059456 5504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:04.059574 5504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:04.059640 5504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:04.098191 5504 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:46487
I20260812 16:45:04.098264 5718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:46487 every 8 connection(s)
I20260812 16:45:04.099619 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:04.108734 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 5504
I20260812 16:45:04.108923 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:45:04.110049 29644 external_mini_cluster.cc:1428] Setting key 0b44dec81b05d84a29261f8ae08aab45
I20260812 16:45:04.111773 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:04.102334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:41989 (local address 127.28.243.62:43407)
0812 16:45:04.102798 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:04.102803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:04.104100 (+ 1297us) server_negotiation.cc:407] Connection header received
0812 16:45:04.107592 (+ 3492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:04.107604 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:04.107760 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:04.107943 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:04.110500 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:04.110518 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:04.110523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:04.110560 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:04.110631 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:04.110655 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:04.110815 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:04.111399 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 16:45:04.111467 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260812 16:45:04.113221 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:04.102955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43407 (local address 127.28.243.1:41989)
0812 16:45:04.103918 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:04.103957 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:04.106377 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:04.108379 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:04.108394 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:04.108553 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:04.109925 (+ 1372us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:04.109930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:04.110008 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:04.110155 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:04.110802 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:04.110816 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:04.110827 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 16:45:04.111618 (+ 791us) client_negotiation.cc:240] Negotiation successful
0812 16:45:04.112088 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
I20260812 16:45:04.113857 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43407
--builtin_ntp_servers=127.28.243.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 16:45:04.115548 5719 heartbeater.cc:347] Connected to a master server at 127.28.243.62:43407
I20260812 16:45:04.115933 5719 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:04.116730 5719 heartbeater.cc:511] Master 127.28.243.62:43407 requested a full tablet report, sending...
I20260812 16:45:04.119206 5400 ts_manager.cc:194] Registered new tserver with Master: f8adb6c6016040de9a2491dc8d2c7e53 (127.28.243.1:46487)
W20260812 16:45:04.536582 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:04.537011 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:04.537148 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:04.545971 5727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:04.546123 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:04.546274 5727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:04.557467 5727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:04.558861 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:04.561141 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:04.578352 5737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:04.578406 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:04.579030 5738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:04.579555 5727 server_base.cc:1034] running on GCE node
I20260812 16:45:04.580231 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:04.581315 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:04.582552 5727 hybrid_clock.cc:648] HybridClock initialized: now 1786553104582502 us; error 44 us; skew 500 ppm
I20260812 16:45:04.582976 5727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:04.585985 5727 webserver.cc:460] Webserver started at http://127.28.243.2:41199/ using document root <none> and password file <none>
I20260812 16:45:04.587136 5727 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:04.587275 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:04.587721 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:04.593014 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "8c94ed122253466b99d6f436b4654db3"
format_stamp: "Formatted at 2026-08-12 16:45:04 on dist-test-slave-1vmg"
server_key: "d99a81702b9351982f16521f77b087de"
server_key_iv: "9e8bbbd4f461236b03e27e381c799c51"
server_key_version: "encryptionkey@0"
I20260812 16:45:04.594241 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "8c94ed122253466b99d6f436b4654db3"
format_stamp: "Formatted at 2026-08-12 16:45:04 on dist-test-slave-1vmg"
server_key: "d99a81702b9351982f16521f77b087de"
server_key_iv: "9e8bbbd4f461236b03e27e381c799c51"
server_key_version: "encryptionkey@0"
I20260812 16:45:04.601505 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 16:45:04.605808 5750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:04.608222 5727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:04.608418 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "8c94ed122253466b99d6f436b4654db3"
format_stamp: "Formatted at 2026-08-12 16:45:04 on dist-test-slave-1vmg"
server_key: "d99a81702b9351982f16521f77b087de"
server_key_iv: "9e8bbbd4f461236b03e27e381c799c51"
server_key_version: "encryptionkey@0"
I20260812 16:45:04.608904 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:04.616029 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:04.617008 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:04.617376 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:04.618436 5727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:04.620086 5727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:04.620193 5727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:04.620309 5727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:04.620388 5727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:04.661348 5727 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:42249
I20260812 16:45:04.661424 5943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:42249 every 8 connection(s)
I20260812 16:45:04.662743 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:04.669126 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 5727
I20260812 16:45:04.669273 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:45:04.669737 29644 external_mini_cluster.cc:1428] Setting key f3b0ab5a01b97bb2053c78355d9aadf4
I20260812 16:45:04.673296 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43407
--builtin_ntp_servers=127.28.243.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 16:45:04.674538 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:04.665665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:33765 (local address 127.28.243.62:43407)
0812 16:45:04.666217 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:04.666285 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:04.667741 (+ 1456us) server_negotiation.cc:407] Connection header received
0812 16:45:04.670748 (+ 3007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:04.670755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:04.670854 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:04.671017 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:04.673193 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:04.673197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:04.673201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:04.673219 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:04.673354 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:04.673390 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:04.673558 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:04.673878 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 16:45:04.674146 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":294,"threads_started":1}
I20260812 16:45:04.676031 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:04.666153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43407 (local address 127.28.243.2:33765)
0812 16:45:04.667461 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:04.667549 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:04.670068 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:04.671467 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:04.671481 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:04.671635 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:04.672683 (+ 1048us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:04.672685 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:04.672735 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:04.672942 (+ 207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:04.673616 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:04.673625 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:04.673633 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:45:04.674159 (+ 526us) client_negotiation.cc:240] Negotiation successful
0812 16:45:04.674665 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
I20260812 16:45:04.677784 5944 heartbeater.cc:347] Connected to a master server at 127.28.243.62:43407
I20260812 16:45:04.678143 5944 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:04.678870 5944 heartbeater.cc:511] Master 127.28.243.62:43407 requested a full tablet report, sending...
I20260812 16:45:04.680792 5400 ts_manager.cc:194] Registered new tserver with Master: 8c94ed122253466b99d6f436b4654db3 (127.28.243.2:42249)
W20260812 16:45:05.048378 5950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:05.048816 5950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:05.048928 5950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:05.059304 5950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:05.059499 5950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:05.059688 5950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:05.070660 5950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:05.072052 5950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:05.073915 5950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:05.089315 5963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:05.089553 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:05.089454 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:05.091197 5950 server_base.cc:1034] running on GCE node
I20260812 16:45:05.092047 5950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:05.093232 5950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:05.094501 5950 hybrid_clock.cc:648] HybridClock initialized: now 1786553105094419 us; error 65 us; skew 500 ppm
I20260812 16:45:05.094874 5950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:05.098059 5950 webserver.cc:460] Webserver started at http://127.28.243.3:44275/ using document root <none> and password file <none>
I20260812 16:45:05.099201 5950 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:05.099370 5950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:05.099884 5950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:05.104605 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "8162283a44ac46c08f693832eca050bc"
format_stamp: "Formatted at 2026-08-12 16:45:05 on dist-test-slave-1vmg"
server_key: "636bca4f06852c2b3b903d5f5364857a"
server_key_iv: "30503c5f1c0ba2c50d1604cda25efd69"
server_key_version: "encryptionkey@0"
I20260812 16:45:05.105846 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "8162283a44ac46c08f693832eca050bc"
format_stamp: "Formatted at 2026-08-12 16:45:05 on dist-test-slave-1vmg"
server_key: "636bca4f06852c2b3b903d5f5364857a"
server_key_iv: "30503c5f1c0ba2c50d1604cda25efd69"
server_key_version: "encryptionkey@0"
I20260812 16:45:05.112488 5950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:45:05.117049 5975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:05.119706 5950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:05.120038 5950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "8162283a44ac46c08f693832eca050bc"
format_stamp: "Formatted at 2026-08-12 16:45:05 on dist-test-slave-1vmg"
server_key: "636bca4f06852c2b3b903d5f5364857a"
server_key_iv: "30503c5f1c0ba2c50d1604cda25efd69"
server_key_version: "encryptionkey@0"
I20260812 16:45:05.120342 5950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:05.122660 5719 heartbeater.cc:503] Master 127.28.243.62:43407 was elected leader, sending a full tablet report...
I20260812 16:45:05.128924 5950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:05.130158 5950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:05.130475 5950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:05.131685 5950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:05.133389 5950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:05.133496 5950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:05.133616 5950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:05.133687 5950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:05.169368 5950 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:45197
I20260812 16:45:05.169546 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:45197 every 8 connection(s)
I20260812 16:45:05.170801 5950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:05.172696 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 5950
I20260812 16:45:05.172890 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:45:05.173475 29644 external_mini_cluster.cc:1428] Setting key 4941e0652caf060111ba1775794eaf50
I20260812 16:45:05.180878 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.174300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:38255 (local address 127.28.243.62:43407)
0812 16:45:05.174480 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.174488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.176355 (+ 1867us) server_negotiation.cc:407] Connection header received
0812 16:45:05.178207 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.178212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.178279 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.178388 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.180036 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.180045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.180049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.180071 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.180107 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.180120 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.180251 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.180515 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.180583 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:45:05.181893 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.175062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43407 (local address 127.28.243.3:38255)
0812 16:45:05.176192 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.176249 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:05.177902 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:05.178659 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:05.178669 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:05.178771 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:05.179599 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:05.179602 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:05.179649 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:05.179775 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:05.180264 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:05.180275 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:05.180283 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:45:05.180553 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 16:45:05.180801 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":354,"threads_started":1}
I20260812 16:45:05.183652 6171 heartbeater.cc:347] Connected to a master server at 127.28.243.62:43407
I20260812 16:45:05.184275 6171 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:05.184989 6171 heartbeater.cc:511] Master 127.28.243.62:43407 requested a full tablet report, sending...
I20260812 16:45:05.186939 5400 ts_manager.cc:194] Registered new tserver with Master: 8162283a44ac46c08f693832eca050bc (127.28.243.3:45197)
I20260812 16:45:05.189155 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:45:05.196995 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.194237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.28.243.62:43407)
0812 16:45:05.194474 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.194480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.194785 (+ 305us) server_negotiation.cc:407] Connection header received
0812 16:45:05.195001 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.195005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.195063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.195156 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.196322 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.196326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.196330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.196347 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.196382 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.196394 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.196518 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.196615 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.196694 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 16:45:05.201593 5399 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:05.208640 5399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:05.249248 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.224369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.28.243.3:45197)
0812 16:45:05.234392 (+ 10023us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.234402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.243892 (+ 9490us) server_negotiation.cc:407] Connection header received
0812 16:45:05.244003 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.244014 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.244668 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.244891 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.247447 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.247461 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.247482 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.247509 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.247611 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.247642 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.247992 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.248663 (+ 671us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.248799 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9815,"thread_start_us":9612,"threads_started":1}
I20260812 16:45:05.249989 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.223295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47398)
0812 16:45:05.235023 (+ 11728us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.235080 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:05.235577 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:05.245096 (+ 9519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:05.245105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:05.245210 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:05.246146 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:05.246149 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:05.246182 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:05.246287 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:05.248680 (+ 2393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:05.248686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:05.248823 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 16:45:05.248985 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 16:45:05.249023 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11381,"thread_start_us":11266,"threads_started":1}
I20260812 16:45:05.250088 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.235910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:42249 (local address 127.0.0.1:54436)
0812 16:45:05.244533 (+ 8623us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.244556 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:05.246509 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:05.247872 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:05.247877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:05.247903 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:05.248572 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:05.248575 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:05.248594 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:05.248643 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:05.249144 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:05.249149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:05.249153 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 16:45:05.249398 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 16:45:05.249448 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8477,"thread_start_us":8338,"threads_started":1}
I20260812 16:45:05.250255 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.236026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.28.243.2:42249)
0812 16:45:05.242976 (+ 6950us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.242989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.245691 (+ 2702us) server_negotiation.cc:407] Connection header received
0812 16:45:05.246976 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.246987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.247245 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.247381 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.248821 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.248829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.248841 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.248860 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.248917 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.248940 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.249048 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.249857 (+ 809us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.249957 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6694,"thread_start_us":6532,"threads_started":1}
I20260812 16:45:05.250335 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.223586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.28.243.1:46487)
0812 16:45:05.244147 (+ 20561us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.244409 (+ 262us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.244450 (+ 41us) server_negotiation.cc:407] Connection header received
0812 16:45:05.244543 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.244551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.244859 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.246363 (+ 1504us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.248019 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.248033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.248052 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.248081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.248167 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.248194 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.249539 (+ 1345us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.249638 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.249750 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20382,"thread_start_us":20221,"threads_started":1}
I20260812 16:45:05.251186 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.223773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:45197 (local address 127.0.0.1:39514)
0812 16:45:05.243368 (+ 19595us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.243392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:05.243545 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:05.246199 (+ 2654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:05.246204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:05.246229 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:05.247013 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:05.247018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:05.247048 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:05.247132 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:05.248094 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:05.248109 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:05.248122 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 16:45:05.248880 (+ 758us) client_negotiation.cc:240] Negotiation successful
0812 16:45:05.249017 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19414,"thread_start_us":20720,"threads_started":1}
I20260812 16:45:05.255473 5836 tablet_service.cc:1467] Processing CreateTablet for tablet c447dd51300b45e6abf9fc555a45ba96 (DEFAULT_TABLE table=test-table [id=60405d6aa55c4237af4aa1e0a2f5ba18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:05.255474 5611 tablet_service.cc:1467] Processing CreateTablet for tablet c447dd51300b45e6abf9fc555a45ba96 (DEFAULT_TABLE table=test-table [id=60405d6aa55c4237af4aa1e0a2f5ba18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:05.257261 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c447dd51300b45e6abf9fc555a45ba96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:05.257261 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c447dd51300b45e6abf9fc555a45ba96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:05.257194 6061 tablet_service.cc:1467] Processing CreateTablet for tablet c447dd51300b45e6abf9fc555a45ba96 (DEFAULT_TABLE table=test-table [id=60405d6aa55c4237af4aa1e0a2f5ba18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:05.259140 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c447dd51300b45e6abf9fc555a45ba96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:05.274261 6202 tablet_bootstrap.cc:492] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Bootstrap starting.
I20260812 16:45:05.277295 6203 tablet_bootstrap.cc:492] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Bootstrap starting.
I20260812 16:45:05.279084 6202 tablet_bootstrap.cc:654] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:05.279104 6204 tablet_bootstrap.cc:492] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Bootstrap starting.
I20260812 16:45:05.280761 6202 log.cc:824] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:05.281512 6203 tablet_bootstrap.cc:654] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:05.282959 6203 log.cc:824] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:05.284351 6204 tablet_bootstrap.cc:654] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:05.285575 6202 tablet_bootstrap.cc:492] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: No bootstrap required, opened a new log
I20260812 16:45:05.285899 6202 ts_tablet_manager.cc:1397] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260812 16:45:05.286048 6204 log.cc:824] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:05.287631 6203 tablet_bootstrap.cc:492] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: No bootstrap required, opened a new log
I20260812 16:45:05.287931 6203 ts_tablet_manager.cc:1397] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260812 16:45:05.288906 6204 tablet_bootstrap.cc:492] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: No bootstrap required, opened a new log
I20260812 16:45:05.289654 6204 ts_tablet_manager.cc:1397] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260812 16:45:05.297411 6202 raft_consensus.cc:348] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.298027 6202 raft_consensus.cc:374] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:05.298202 6202 raft_consensus.cc:729] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c94ed122253466b99d6f436b4654db3, State: Initialized, Role: FOLLOWER
I20260812 16:45:05.298702 6203 raft_consensus.cc:348] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.299222 6203 raft_consensus.cc:374] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:05.299101 6202 consensus_queue.cc:260] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.299324 6203 raft_consensus.cc:729] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adb6c6016040de9a2491dc8d2c7e53, State: Initialized, Role: FOLLOWER
I20260812 16:45:05.299939 6204 raft_consensus.cc:348] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.300001 6203 consensus_queue.cc:260] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.300302 6204 raft_consensus.cc:374] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:05.300432 6204 raft_consensus.cc:729] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8162283a44ac46c08f693832eca050bc, State: Initialized, Role: FOLLOWER
I20260812 16:45:05.301146 6204 consensus_queue.cc:260] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.302047 6203 ts_tablet_manager.cc:1428] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 16:45:05.302230 6171 heartbeater.cc:503] Master 127.28.243.62:43407 was elected leader, sending a full tablet report...
I20260812 16:45:05.302850 6204 ts_tablet_manager.cc:1428] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260812 16:45:05.303042 5944 heartbeater.cc:503] Master 127.28.243.62:43407 was elected leader, sending a full tablet report...
I20260812 16:45:05.304088 6202 ts_tablet_manager.cc:1428] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
W20260812 16:45:05.352279 5720 tablet.cc:2369] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:45:05.415246 5945 tablet.cc:2369] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:45:05.422816 6173 tablet.cc:2369] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:05.476151 6216 raft_consensus.cc:482] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:05.476498 6216 raft_consensus.cc:504] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.478406 6216 leader_election.cc:288] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8162283a44ac46c08f693832eca050bc (127.28.243.3:45197), 8c94ed122253466b99d6f436b4654db3 (127.28.243.2:42249)
I20260812 16:45:05.481827 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.478982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:41551 (local address 127.28.243.3:45197)
0812 16:45:05.479232 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.479240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.479888 (+ 648us) server_negotiation.cc:407] Connection header received
0812 16:45:05.480071 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.480075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.480129 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.480214 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.481211 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.481217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.481222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.481241 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.481277 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.481289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.481405 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.481482 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.481549 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:45:05.482115 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.478714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:45197 (local address 127.28.243.1:41551)
0812 16:45:05.479772 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.479791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:05.479940 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:05.480271 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:05.480276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:05.480300 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:05.480995 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:05.480999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:05.481018 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:05.481070 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:05.481375 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:05.481380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:05.481385 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 16:45:05.481540 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 16:45:05.481593 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":699,"threads_started":1}
I20260812 16:45:05.482432 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.479374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:36165 (local address 127.28.243.2:42249)
0812 16:45:05.479586 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.479591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.480197 (+ 606us) server_negotiation.cc:407] Connection header received
0812 16:45:05.480356 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.480361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.480415 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.480501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.481617 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.481622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.481626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.481652 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.481696 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.481710 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.481918 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.482090 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.482166 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:45:05.482635 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.479107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:42249 (local address 127.28.243.1:36165)
0812 16:45:05.480082 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.480097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:05.480229 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:05.480555 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:05.480560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:05.480584 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:05.481236 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:45:05.481240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:05.481265 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:45:05.481340 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:05.481823 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:05.481905 (+ 82us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:05.481920 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 16:45:05.482153 (+ 233us) client_negotiation.cc:240] Negotiation successful
0812 16:45:05.482213 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":674,"threads_started":1}
I20260812 16:45:05.483616 6092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c447dd51300b45e6abf9fc555a45ba96" candidate_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8162283a44ac46c08f693832eca050bc" is_pre_election: true
I20260812 16:45:05.483794 5874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c447dd51300b45e6abf9fc555a45ba96" candidate_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c94ed122253466b99d6f436b4654db3" is_pre_election: true
I20260812 16:45:05.484190 6092 raft_consensus.cc:2454] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8adb6c6016040de9a2491dc8d2c7e53 in term 0.
I20260812 16:45:05.484411 5874 raft_consensus.cc:2454] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8adb6c6016040de9a2491dc8d2c7e53 in term 0.
I20260812 16:45:05.484956 5541 leader_election.cc:302] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8162283a44ac46c08f693832eca050bc, f8adb6c6016040de9a2491dc8d2c7e53; no voters:
I20260812 16:45:05.485426 6216 raft_consensus.cc:2781] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:05.485571 6216 raft_consensus.cc:482] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:05.485671 6216 raft_consensus.cc:3037] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:05.487962 6216 raft_consensus.cc:504] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.488924 6216 leader_election.cc:288] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [CANDIDATE]: Term 1 election: Requested vote from peers 8162283a44ac46c08f693832eca050bc (127.28.243.3:45197), 8c94ed122253466b99d6f436b4654db3 (127.28.243.2:42249)
I20260812 16:45:05.489280 6092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c447dd51300b45e6abf9fc555a45ba96" candidate_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8162283a44ac46c08f693832eca050bc"
I20260812 16:45:05.489521 6092 raft_consensus.cc:3037] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:05.489504 5874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c447dd51300b45e6abf9fc555a45ba96" candidate_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c94ed122253466b99d6f436b4654db3"
I20260812 16:45:05.489720 5874 raft_consensus.cc:3037] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:05.492964 6092 raft_consensus.cc:2454] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8adb6c6016040de9a2491dc8d2c7e53 in term 1.
I20260812 16:45:05.492964 5874 raft_consensus.cc:2454] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8adb6c6016040de9a2491dc8d2c7e53 in term 1.
I20260812 16:45:05.493641 5538 leader_election.cc:302] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c94ed122253466b99d6f436b4654db3, f8adb6c6016040de9a2491dc8d2c7e53; no voters:
I20260812 16:45:05.494058 6216 raft_consensus.cc:2781] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:05.494812 6216 raft_consensus.cc:686] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 1 LEADER]: Becoming Leader. State: Replica: f8adb6c6016040de9a2491dc8d2c7e53, State: Running, Role: LEADER
I20260812 16:45:05.495482 6216 consensus_queue.cc:237] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } }
I20260812 16:45:05.501049 5400 catalog_manager.cc:5495] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8adb6c6016040de9a2491dc8d2c7e53 (127.28.243.1). New cstate: current_term: 1 leader_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8adb6c6016040de9a2491dc8d2c7e53" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 46487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 } health_report { overall_health: UNKNOWN } } }
I20260812 16:45:05.555060 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.552397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.28.243.1:46487)
0812 16:45:05.552661 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.552666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.552688 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:45:05.552865 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.552869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.552960 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.553050 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.554338 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.554343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.554346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.554362 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.554397 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.554410 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.554533 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.554691 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.554758 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:45:05.563861 5874 raft_consensus.cc:1261] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 1 FOLLOWER]: Refusing update from remote peer f8adb6c6016040de9a2491dc8d2c7e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:05.563861 6092 raft_consensus.cc:1261] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 1 FOLLOWER]: Refusing update from remote peer f8adb6c6016040de9a2491dc8d2c7e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:05.564882 6216 consensus_queue.cc:1035] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8162283a44ac46c08f693832eca050bc" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:05.565371 6225 consensus_queue.cc:1035] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c94ed122253466b99d6f436b4654db3" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 16:45:05.601641 5400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:05.601912 5400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:05.606518 5400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0f65a5cb2c4ad7a5e747faf2c73990: Sending DeleteTablet for 3 replicas of tablet c447dd51300b45e6abf9fc555a45ba96
I20260812 16:45:05.608422 5611 tablet_service.cc:1514] Processing DeleteTablet for tablet c447dd51300b45e6abf9fc555a45ba96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:05 UTC) from {username='slave'} at 127.0.0.1:47398
I20260812 16:45:05.609313 6061 tablet_service.cc:1514] Processing DeleteTablet for tablet c447dd51300b45e6abf9fc555a45ba96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:05 UTC) from {username='slave'} at 127.0.0.1:39514
I20260812 16:45:05.609478 5836 tablet_service.cc:1514] Processing DeleteTablet for tablet c447dd51300b45e6abf9fc555a45ba96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:05 UTC) from {username='slave'} at 127.0.0.1:54436
I20260812 16:45:05.610293 6246 tablet_replica.cc:330] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: stopping tablet replica
I20260812 16:45:05.611726 6246 raft_consensus.cc:2229] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:05.612001 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 5504
I20260812 16:45:05.612416 6246 raft_consensus.cc:2258] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:05.615363 6244 tablet_replica.cc:330] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: stopping tablet replica
I20260812 16:45:05.615897 6244 raft_consensus.cc:2229] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:05.616716 6246 ts_tablet_manager.cc:1905] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:05.616739 6244 raft_consensus.cc:2258] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:05.624369 6245 tablet_replica.cc:330] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: stopping tablet replica
I20260812 16:45:05.624598 6246 ts_tablet_manager.cc:1918] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:05.624794 6246 log.cc:1196] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/c447dd51300b45e6abf9fc555a45ba96
I20260812 16:45:05.625334 6245 raft_consensus.cc:2229] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:05.625918 6246 ts_tablet_manager.cc:1939] T c447dd51300b45e6abf9fc555a45ba96 P 8c94ed122253466b99d6f436b4654db3: Deleting consensus metadata
I20260812 16:45:05.626096 6245 raft_consensus.cc:2258] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:05.628054 6244 ts_tablet_manager.cc:1905] T c447dd51300b45e6abf9fc555a45ba96 P f8adb6c6016040de9a2491dc8d2c7e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:05.628197 5371 catalog_manager.cc:4841] TS 8c94ed122253466b99d6f436b4654db3 (127.28.243.2:42249): tablet c447dd51300b45e6abf9fc555a45ba96 (table test-table [id=60405d6aa55c4237af4aa1e0a2f5ba18]) successfully deleted
I20260812 16:45:05.628399 6245 ts_tablet_manager.cc:1905] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:05.631217 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.614441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.28.243.1:46487)
0812 16:45:05.614629 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.614636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.624432 (+ 9796us) server_negotiation.cc:407] Connection header received
0812 16:45:05.624636 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.624642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.624715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.624843 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.630147 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.630158 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.630163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.630188 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.630238 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.630256 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.630429 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.630735 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.630817 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 16:45:05.632762 5712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:05.635872 6245 ts_tablet_manager.cc:1918] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:05.636072 6245 log.cc:1196] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/c447dd51300b45e6abf9fc555a45ba96
I20260812 16:45:05.636698 6245 ts_tablet_manager.cc:1939] T c447dd51300b45e6abf9fc555a45ba96 P 8162283a44ac46c08f693832eca050bc: Deleting consensus metadata
I20260812 16:45:05.638441 5376 catalog_manager.cc:4841] TS 8162283a44ac46c08f693832eca050bc (127.28.243.3:45197): tablet c447dd51300b45e6abf9fc555a45ba96 (table test-table [id=60405d6aa55c4237af4aa1e0a2f5ba18]) successfully deleted
I20260812 16:45:05.736430 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 5504
I20260812 16:45:05.768281 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 5727
W20260812 16:45:05.768502 5371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:46487 (error 108)
I20260812 16:45:05.770782 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.769967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47414)
0812 16:45:05.770448 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.770571 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":245,"threads_started":1}
W20260812 16:45:05.771150 5371 catalog_manager.cc:4584] TS f8adb6c6016040de9a2491dc8d2c7e53 (127.28.243.1:46487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c447dd51300b45e6abf9fc555a45ba96: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
I20260812 16:45:05.772511 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.769336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.28.243.2:42249)
0812 16:45:05.769613 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.769619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.769648 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:45:05.769873 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.769878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.770074 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.770208 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.771569 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.771574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.771578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.771599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.771640 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.771653 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.771794 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.772039 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.772154 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 16:45:05.773345 5936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:05.825124 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.824645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47424)
0812 16:45:05.824867 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.824968 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 16:45:05.826735 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.826354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47434)
0812 16:45:05.826513 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.826587 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 16:45:05.862952 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 5727
I20260812 16:45:05.887826 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.887249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47436)
0812 16:45:05.887506 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.887629 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 16:45:05.888976 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.888575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47450)
0812 16:45:05.888727 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.888821 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 16:45:05.891525 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 5950
I20260812 16:45:05.895295 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.892596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.28.243.3:45197)
0812 16:45:05.892789 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:05.892794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:05.892815 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:45:05.893058 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:05.893065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:05.893209 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:05.893342 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:05.894491 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:05.894496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:05.894500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:05.894519 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:05.894557 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:05.894571 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:05.894701 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:05.894907 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 16:45:05.895037 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:45:05.896376 6164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:05.970445 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.969876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47458)
0812 16:45:05.970185 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.970282 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 16:45:05.972292 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:05.971714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:46487 (local address 127.0.0.1:47460)
0812 16:45:05.972019 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:05.972123 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260812 16:45:05.988761 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 5950
I20260812 16:45:06.013276 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 5349
I20260812 16:45:06.014645 5453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:06.126770 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 5349
2026-08-12T16:45:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3309 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 16:45:06.188184 29644 mini_oidc.cc:152] Starting JWKS server
I20260812 16:45:06.193037 29644 webserver.cc:460] Webserver started at https://127.0.0.1:37853/ using document root <none> and password file <none>
I20260812 16:45:06.193655 29644 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 16:45:06.194706 29644 webserver.cc:460] Webserver started at http://127.0.0.1:37157/ using document root <none> and password file <none>
2026-08-12T16:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:06Z Disabled control of system clock
I20260812 16:45:06.234479 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36609
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:35455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:36609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37853/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/testchainca.cert with env {}
W20260812 16:45:06.609481 6270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:06.609879 6270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:06.610005 6270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:06.618322 6270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:06.618568 6270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:06.618635 6270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:06.618732 6270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:06.618801 6270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:45:06.618945 6270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:45:06.619002 6270 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37853/jwks.json
I20260812 16:45:06.631460 6270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:35455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:36609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37853/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:06.633622 6270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:06.635673 6270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:06.649643 6280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:06.649525 6283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:06.649925 6281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:06.650805 6270 server_base.cc:1034] running on GCE node
I20260812 16:45:06.651785 6270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:06.653400 6270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:06.654672 6270 hybrid_clock.cc:648] HybridClock initialized: now 1786553106654610 us; error 55 us; skew 500 ppm
I20260812 16:45:06.655180 6270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:06.657815 6270 webserver.cc:460] Webserver started at http://127.28.243.62:37413/ using document root <none> and password file <none>
I20260812 16:45:06.658594 6270 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:06.658718 6270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:06.659117 6270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:06.662480 6270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "fb9b656b854548d4be1316e3eedabd2b"
format_stamp: "Formatted at 2026-08-12 16:45:06 on dist-test-slave-1vmg"
server_key: "aa2e39038858bd04146f13af0af71031"
server_key_iv: "f98b00b07c8a6280cbb35446d4e5b64c"
server_key_version: "encryptionkey@0"
I20260812 16:45:06.663429 6270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "fb9b656b854548d4be1316e3eedabd2b"
format_stamp: "Formatted at 2026-08-12 16:45:06 on dist-test-slave-1vmg"
server_key: "aa2e39038858bd04146f13af0af71031"
server_key_iv: "f98b00b07c8a6280cbb35446d4e5b64c"
server_key_version: "encryptionkey@0"
I20260812 16:45:06.669909 6270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:45:06.674202 6298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:06.676409 6270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 16:45:06.676719 6270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "fb9b656b854548d4be1316e3eedabd2b"
format_stamp: "Formatted at 2026-08-12 16:45:06 on dist-test-slave-1vmg"
server_key: "aa2e39038858bd04146f13af0af71031"
server_key_iv: "f98b00b07c8a6280cbb35446d4e5b64c"
server_key_version: "encryptionkey@0"
I20260812 16:45:06.677042 6270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:06.713109 6270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:06.714056 6270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:06.714394 6270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:06.736649 6270 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:36609
I20260812 16:45:06.736654 6385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:36609 every 8 connection(s)
I20260812 16:45:06.738134 6270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:06.742422 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:06.745981 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 6270
I20260812 16:45:06.746243 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:06.747411 29644 external_mini_cluster.cc:1428] Setting key 80041329a272972e3e45398520dd3a1b
I20260812 16:45:06.759581 6386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b: Bootstrap starting.
I20260812 16:45:06.764061 6386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:06.765354 6386 log.cc:824] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:06.765574 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:06.750622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.28.243.62:36609)
0812 16:45:06.751950 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:06.751963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:06.752016 (+ 53us) server_negotiation.cc:407] Connection header received
0812 16:45:06.754057 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:06.754086 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:06.754867 (+ 781us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:06.755543 (+ 676us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:06.756438 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:06.757928 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:06.759923 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:06.760674 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:06.760831 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:06.762344 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:06.762353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:06.762370 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:06.763211 (+ 841us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:06.763411 (+ 200us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:06.763459 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:06.763640 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:06.764002 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 16:45:06.764445 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":473,"threads_started":1}
I20260812 16:45:06.770185 6386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b: No bootstrap required, opened a new log
I20260812 16:45:06.771695 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:45:06.780165 6386 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9b656b854548d4be1316e3eedabd2b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36609 } }
I20260812 16:45:06.780722 6386 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:06.780840 6386 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9b656b854548d4be1316e3eedabd2b, State: Initialized, Role: FOLLOWER
I20260812 16:45:06.781484 6386 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9b656b854548d4be1316e3eedabd2b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36609 } }
I20260812 16:45:06.781736 6386 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:06.781867 6386 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:06.782025 6386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:06.784802 6386 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9b656b854548d4be1316e3eedabd2b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36609 } }
I20260812 16:45:06.785881 6386 leader_election.cc:302] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb9b656b854548d4be1316e3eedabd2b; no voters:
I20260812 16:45:06.786580 6386 leader_election.cc:288] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:06.786712 6394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:06.787901 6394 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [term 1 LEADER]: Becoming Leader. State: Replica: fb9b656b854548d4be1316e3eedabd2b, State: Running, Role: LEADER
I20260812 16:45:06.788545 6394 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9b656b854548d4be1316e3eedabd2b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36609 } }
I20260812 16:45:06.788902 6386 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:06.791467 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb9b656b854548d4be1316e3eedabd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9b656b854548d4be1316e3eedabd2b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36609 } } }
I20260812 16:45:06.792557 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:06.792476 6396 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9b656b854548d4be1316e3eedabd2b. Latest consensus state: current_term: 1 leader_uuid: "fb9b656b854548d4be1316e3eedabd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9b656b854548d4be1316e3eedabd2b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36609 } } }
I20260812 16:45:06.793269 6396 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:06.795745 6405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:06.801613 6405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:06.814194 6405 catalog_manager.cc:1347] Generated new cluster ID: a5209f1735214261a642d6fb29d51a5d
I20260812 16:45:06.814409 6405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:06.834969 6405 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:06.836949 6405 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:06.851239 6405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb9b656b854548d4be1316e3eedabd2b: Generated new TSK 0
I20260812 16:45:06.852596 6405 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 16:45:06.897449 6430 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 16:45:06.897701 6430 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 16:45:06.900486 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:06.875198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.28.243.62:36609)
0812 16:45:06.875378 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:06.875383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:06.876204 (+ 821us) server_negotiation.cc:407] Connection header received
0812 16:45:06.876716 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:06.876721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:06.876805 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:06.876963 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:45:06.877570 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:06.878366 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:06.879885 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:06.880337 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:06.880448 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:06.881036 (+ 588us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:45:06.899710 (+ 18674us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 16:45:06.900150 (+ 440us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 16:45:06.900640 6389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 16:45:06.902503 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:06.875296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36609 (local address 127.0.0.1:44636)
0812 16:45:06.876056 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:06.876075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:06.876303 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:06.876996 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:06.877001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:06.877014 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:45:06.877393 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:06.877403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:06.878509 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:06.878513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:06.879718 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:06.879739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:06.880527 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:06.880533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:06.880672 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:06.880749 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:45:06.900105 (+ 19356us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 16:45:06.900507 (+ 402us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.62:36609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":400,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.243.62:36609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 16:45:06.904521 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 6270
I20260812 16:45:06.905812 6379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:07.009027 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 6270
2026-08-12T16:45:07Z chronyd exiting
I20260812 16:45:07.078869 29644 test_util.cc:175] -----------------------------------------------
I20260812 16:45:07.079089 29644 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786553056142401-29644-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (897 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 16:45:07.083216 29644 mini_oidc.cc:152] Starting JWKS server
I20260812 16:45:07.085198 29644 webserver.cc:460] Webserver started at https://127.0.0.1:38741/ using document root <none> and password file <none>
I20260812 16:45:07.085741 29644 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 16:45:07.086510 29644 webserver.cc:460] Webserver started at http://127.0.0.1:41259/ using document root <none> and password file <none>
2026-08-12T16:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:07Z Disabled control of system clock
I20260812 16:45:07.127828 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:34865
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:34865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38741/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/testchainca.cert with env {}
W20260812 16:45:07.505242 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:07.505910 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:07.506114 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:07.514530 6441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:07.514696 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:07.514767 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:07.514856 6441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:07.514948 6441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:45:07.515041 6441 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:45:07.515095 6441 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38741/jwks.json
I20260812 16:45:07.527415 6441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:34865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38741/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:07.528947 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:07.530514 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:07.544304 6447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:07.544353 6448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:07.544273 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:07.545213 6441 server_base.cc:1034] running on GCE node
I20260812 16:45:07.546176 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:07.547749 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:07.548965 6441 hybrid_clock.cc:648] HybridClock initialized: now 1786553107548918 us; error 37 us; skew 500 ppm
I20260812 16:45:07.549350 6441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:07.553937 6441 webserver.cc:460] Webserver started at http://127.28.243.62:35185/ using document root <none> and password file <none>
I20260812 16:45:07.555037 6441 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:07.555186 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:07.555603 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:07.559554 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "00061565f2f94a8ab97c911606768982"
format_stamp: "Formatted at 2026-08-12 16:45:07 on dist-test-slave-1vmg"
server_key: "3c098fe65723ab454ae8a1cc937dfbc8"
server_key_iv: "b78867f303c284a82dfaf424e4ee91a2"
server_key_version: "encryptionkey@0"
I20260812 16:45:07.560590 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "00061565f2f94a8ab97c911606768982"
format_stamp: "Formatted at 2026-08-12 16:45:07 on dist-test-slave-1vmg"
server_key: "3c098fe65723ab454ae8a1cc937dfbc8"
server_key_iv: "b78867f303c284a82dfaf424e4ee91a2"
server_key_version: "encryptionkey@0"
I20260812 16:45:07.567943 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:45:07.572530 6466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:07.574407 6441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:45:07.574630 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "00061565f2f94a8ab97c911606768982"
format_stamp: "Formatted at 2026-08-12 16:45:07 on dist-test-slave-1vmg"
server_key: "3c098fe65723ab454ae8a1cc937dfbc8"
server_key_iv: "b78867f303c284a82dfaf424e4ee91a2"
server_key_version: "encryptionkey@0"
I20260812 16:45:07.574891 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:07.600461 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:07.601439 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:07.601802 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:07.622242 6441 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:34865
I20260812 16:45:07.622251 6552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:34865 every 8 connection(s)
I20260812 16:45:07.623808 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:07.628320 6553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:07.633272 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 6441
I20260812 16:45:07.633476 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:07.634335 29644 external_mini_cluster.cc:1428] Setting key 1623a5cc7d09816f60c28be6b957d1e2
I20260812 16:45:07.641383 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982: Bootstrap starting.
I20260812 16:45:07.647143 6553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:07.648768 6553 log.cc:824] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:07.653147 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982: No bootstrap required, opened a new log
I20260812 16:45:07.655023 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:07.636830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.28.243.62:34865)
0812 16:45:07.638061 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:07.638074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:07.638136 (+ 62us) server_negotiation.cc:407] Connection header received
0812 16:45:07.640211 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:07.640247 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:07.640880 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:07.641310 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:07.642363 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:07.644013 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:07.646393 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:07.647714 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:07.647958 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:07.649868 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:07.649896 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:07.649931 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:07.650856 (+ 925us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:07.651021 (+ 165us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:07.651054 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:07.651176 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:07.651898 (+ 722us) server_negotiation.cc:299] Negotiation successful
0812 16:45:07.653138 (+ 1240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":436,"threads_started":1}
I20260812 16:45:07.659794 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:45:07.662307 6553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00061565f2f94a8ab97c911606768982" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34865 } }
I20260812 16:45:07.662796 6553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:07.662899 6553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00061565f2f94a8ab97c911606768982, State: Initialized, Role: FOLLOWER
I20260812 16:45:07.663694 6553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00061565f2f94a8ab97c911606768982" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34865 } }
I20260812 16:45:07.663960 6553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:07.664099 6553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:07.664274 6553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:07.666746 6553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00061565f2f94a8ab97c911606768982" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34865 } }
I20260812 16:45:07.667373 6553 leader_election.cc:302] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00061565f2f94a8ab97c911606768982; no voters:
I20260812 16:45:07.668177 6553 leader_election.cc:288] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:07.668344 6561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:07.669694 6561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [term 1 LEADER]: Becoming Leader. State: Replica: 00061565f2f94a8ab97c911606768982, State: Running, Role: LEADER
I20260812 16:45:07.670507 6561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00061565f2f94a8ab97c911606768982" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34865 } }
I20260812 16:45:07.670989 6553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:07.675182 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00061565f2f94a8ab97c911606768982. Latest consensus state: current_term: 1 leader_uuid: "00061565f2f94a8ab97c911606768982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00061565f2f94a8ab97c911606768982" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34865 } } }
I20260812 16:45:07.675701 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:07.674659 6562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00061565f2f94a8ab97c911606768982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00061565f2f94a8ab97c911606768982" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34865 } } }
I20260812 16:45:07.676617 6562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:07.676817 6575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:07.683063 6575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:07.696157 6575 catalog_manager.cc:1347] Generated new cluster ID: faf596b33f11446fbd52a4172ca477f3
I20260812 16:45:07.696305 6575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:07.719810 6575 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:07.721241 6575 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:07.731730 6575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00061565f2f94a8ab97c911606768982: Generated new TSK 0
I20260812 16:45:07.734031 6575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 16:45:07.778447 6605 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 16:45:07.778599 6605 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 16:45:07.779956 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:07.757733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.28.243.62:34865)
0812 16:45:07.758172 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:07.758179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:07.758551 (+ 372us) server_negotiation.cc:407] Connection header received
0812 16:45:07.758872 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:07.758877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:07.759024 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:07.759210 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:45:07.759850 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:07.760648 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:07.763024 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:07.763599 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:07.763738 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:07.764494 (+ 756us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:45:07.779295 (+ 14801us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 16:45:07.779710 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":266}
W20260812 16:45:07.780093 6558 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 16:45:07.780858 6601 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:07.757467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34865 (local address 127.0.0.1:48218)
0812 16:45:07.758352 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:07.758377 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:07.758587 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:07.759244 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:07.759250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:07.759265 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:45:07.759637 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:07.759651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:07.760812 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:07.760817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:07.762611 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:07.762630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:07.763823 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:07.763830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:07.764060 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:07.764117 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:45:07.779606 (+ 15489us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 16:45:07.779973 (+ 367us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.62:34865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":494,"threads_started":1}
I20260812 16:45:07.783062 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 6441
I20260812 16:45:07.784811 6544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:07.882436 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 6441
2026-08-12T16:45:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (871 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 16:45:07.954370 29644 mini_oidc.cc:152] Starting JWKS server
I20260812 16:45:07.956589 29644 webserver.cc:460] Webserver started at https://127.0.0.1:38105/ using document root <none> and password file <none>
I20260812 16:45:07.956876 29644 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 16:45:07.957705 29644 webserver.cc:460] Webserver started at http://127.0.0.1:39533/ using document root <none> and password file <none>
2026-08-12T16:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:07Z Disabled control of system clock
I20260812 16:45:08.001134 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42421
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:42421
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38105/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 16:45:08.425544 6620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:08.426252 6620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:08.426682 6620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:08.437511 6620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:08.437704 6620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:08.437765 6620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:08.437848 6620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:08.437909 6620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:45:08.438005 6620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:45:08.438058 6620 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38105/jwks.json
I20260812 16:45:08.451148 6620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:42421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38105/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:08.453176 6620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:08.456048 6620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:08.470290 6628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:08.470553 6627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:08.471104 6634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:08.471730 6620 server_base.cc:1034] running on GCE node
I20260812 16:45:08.472625 6620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:08.474263 6620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:08.475478 6620 hybrid_clock.cc:648] HybridClock initialized: now 1786553108475415 us; error 51 us; skew 500 ppm
I20260812 16:45:08.475893 6620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:08.478575 6620 webserver.cc:460] Webserver started at http://127.28.243.62:32953/ using document root <none> and password file <none>
I20260812 16:45:08.479435 6620 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:08.479543 6620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:08.479905 6620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:08.483143 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "65919352f8ab49c083a7f555c8b58411"
format_stamp: "Formatted at 2026-08-12 16:45:08 on dist-test-slave-1vmg"
server_key: "c45a99f53b6e083706dc7032271bd75a"
server_key_iv: "3b90eb635ef945a0b0a759a2d4369045"
server_key_version: "encryptionkey@0"
I20260812 16:45:08.483989 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "65919352f8ab49c083a7f555c8b58411"
format_stamp: "Formatted at 2026-08-12 16:45:08 on dist-test-slave-1vmg"
server_key: "c45a99f53b6e083706dc7032271bd75a"
server_key_iv: "3b90eb635ef945a0b0a759a2d4369045"
server_key_version: "encryptionkey@0"
I20260812 16:45:08.490262 6620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:45:08.496680 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:08.498477 6620 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260812 16:45:08.498710 6620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "65919352f8ab49c083a7f555c8b58411"
format_stamp: "Formatted at 2026-08-12 16:45:08 on dist-test-slave-1vmg"
server_key: "c45a99f53b6e083706dc7032271bd75a"
server_key_iv: "3b90eb635ef945a0b0a759a2d4369045"
server_key_version: "encryptionkey@0"
I20260812 16:45:08.498994 6620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:08.517771 6620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:08.518970 6620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:08.519346 6620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:08.542037 6620 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:42421
I20260812 16:45:08.542062 6723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:42421 every 8 connection(s)
I20260812 16:45:08.543604 6620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:08.548787 6725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:08.554382 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 6620
I20260812 16:45:08.554745 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:08.555639 29644 external_mini_cluster.cc:1428] Setting key ee70b3df1144221d2cf65a180d31fd70
I20260812 16:45:08.565109 6725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411: Bootstrap starting.
I20260812 16:45:08.568929 6725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:08.570612 6725 log.cc:824] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:08.571285 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:08.558050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.28.243.62:42421)
0812 16:45:08.559311 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:08.559323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:08.559373 (+ 50us) server_negotiation.cc:407] Connection header received
0812 16:45:08.561102 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:08.561131 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:08.561746 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:08.562248 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:08.563014 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:08.564366 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:08.565898 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:08.566660 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:08.566901 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:08.568188 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:08.568198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:08.568218 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:08.568815 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:08.568929 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:08.568954 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:08.569089 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:08.569338 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 16:45:08.569706 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":452,"threads_started":1}
I20260812 16:45:08.575601 6725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411: No bootstrap required, opened a new log
I20260812 16:45:08.577406 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:45:08.585647 6725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65919352f8ab49c083a7f555c8b58411" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42421 } }
I20260812 16:45:08.586215 6725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:08.586328 6725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65919352f8ab49c083a7f555c8b58411, State: Initialized, Role: FOLLOWER
I20260812 16:45:08.587255 6725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65919352f8ab49c083a7f555c8b58411" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42421 } }
I20260812 16:45:08.587666 6725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:08.587850 6725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:08.588078 6725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:08.591468 6725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65919352f8ab49c083a7f555c8b58411" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42421 } }
I20260812 16:45:08.592180 6725 leader_election.cc:302] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65919352f8ab49c083a7f555c8b58411; no voters:
I20260812 16:45:08.592985 6725 leader_election.cc:288] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:08.593133 6732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:08.594198 6732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [term 1 LEADER]: Becoming Leader. State: Replica: 65919352f8ab49c083a7f555c8b58411, State: Running, Role: LEADER
I20260812 16:45:08.595039 6732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65919352f8ab49c083a7f555c8b58411" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42421 } }
I20260812 16:45:08.595516 6725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:08.599421 6736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65919352f8ab49c083a7f555c8b58411. Latest consensus state: current_term: 1 leader_uuid: "65919352f8ab49c083a7f555c8b58411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65919352f8ab49c083a7f555c8b58411" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42421 } } }
I20260812 16:45:08.599408 6733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65919352f8ab49c083a7f555c8b58411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65919352f8ab49c083a7f555c8b58411" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42421 } } }
I20260812 16:45:08.599849 6736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:08.599901 6733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:08.601903 6746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:08.608913 6746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:08.620059 6746 catalog_manager.cc:1347] Generated new cluster ID: fd79f3fd573444a88d6e22c89e59e5f6
I20260812 16:45:08.620200 6746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:08.638476 6746 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:08.639997 6746 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:08.655905 6746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65919352f8ab49c083a7f555c8b58411: Generated new TSK 0
I20260812 16:45:08.656948 6746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:45:08.686069 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:08.676883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.28.243.62:42421)
0812 16:45:08.677103 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:08.677110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:08.677392 (+ 282us) server_negotiation.cc:407] Connection header received
0812 16:45:08.677620 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:08.677634 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:08.677737 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:08.678033 (+ 296us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:45:08.678584 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:08.679792 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:08.681288 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:08.681979 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:08.682126 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:08.682768 (+ 642us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:45:08.685360 (+ 2592us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 16:45:08.685731 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":86}
W20260812 16:45:08.686209 6729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 16:45:08.687649 6778 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:08.676664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42421 (local address 127.0.0.1:39866)
0812 16:45:08.677275 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:08.677295 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:08.677436 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:08.678080 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:08.678084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:08.678096 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:45:08.678417 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:08.678426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:08.679990 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:08.679994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:08.681131 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:08.681145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:08.682229 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:08.682235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:08.682391 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:08.682431 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:45:08.685613 (+ 3182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 16:45:08.687229 (+ 1616us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.62:42421: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":1272,"thread_start_us":344,"threads_started":1}
I20260812 16:45:08.688649 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 6620
I20260812 16:45:08.689986 6713 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 16:45:08.697325 6779 webserver.cc:533] Webserver: sslize failed:
E20260812 16:45:08.697425 6779 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 16:45:08.790655 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 6620
2026-08-12T16:45:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (902 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T16:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:08Z Disabled control of system clock
I20260812 16:45:08.899624 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45281
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:45281
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/testchainca.cert with env {}
W20260812 16:45:09.320063 6795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:09.320415 6795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:09.320508 6795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:09.328822 6795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:09.328981 6795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:09.329051 6795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:09.329136 6795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:09.329190 6795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:45:09.329283 6795 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:45:09.329349 6795 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 16:45:09.343006 6795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:45281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:09.344501 6795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:09.346117 6795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:09.361217 6801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:09.361279 6800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:09.361408 6803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:09.361668 6795 server_base.cc:1034] running on GCE node
I20260812 16:45:09.362839 6795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:09.364655 6795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:09.365928 6795 hybrid_clock.cc:648] HybridClock initialized: now 1786553109365858 us; error 61 us; skew 500 ppm
I20260812 16:45:09.366654 6795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:45:09.370426 6795 webserver.cc:460] Webserver started at http://127.28.243.62:36619/ using document root <none> and password file <none>
I20260812 16:45:09.371709 6795 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:09.371922 6795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:09.372284 6795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:09.375792 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "1884e39f81bd4ac78062ca9ff8407521"
format_stamp: "Formatted at 2026-08-12 16:45:09 on dist-test-slave-1vmg"
server_key: "109a6992e44075ee73969acd7d099420"
server_key_iv: "f15e45d0c9de0eb0fa000734586b6da6"
server_key_version: "encryptionkey@0"
I20260812 16:45:09.376717 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "1884e39f81bd4ac78062ca9ff8407521"
format_stamp: "Formatted at 2026-08-12 16:45:09 on dist-test-slave-1vmg"
server_key: "109a6992e44075ee73969acd7d099420"
server_key_iv: "f15e45d0c9de0eb0fa000734586b6da6"
server_key_version: "encryptionkey@0"
I20260812 16:45:09.382673 6795 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 16:45:09.387104 6811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:09.389501 6795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:45:09.389833 6795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "1884e39f81bd4ac78062ca9ff8407521"
format_stamp: "Formatted at 2026-08-12 16:45:09 on dist-test-slave-1vmg"
server_key: "109a6992e44075ee73969acd7d099420"
server_key_iv: "f15e45d0c9de0eb0fa000734586b6da6"
server_key_version: "encryptionkey@0"
I20260812 16:45:09.390146 6795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:09.409088 6795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:09.410324 6795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:09.410683 6795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:09.433131 6795 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:45281
I20260812 16:45:09.433125 6897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:45281 every 8 connection(s)
I20260812 16:45:09.434952 6795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:09.435680 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 6795
I20260812 16:45:09.435889 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:09.436689 29644 external_mini_cluster.cc:1428] Setting key 3ab043b8ce6a5fc459bcb0e75723be0a
I20260812 16:45:09.440127 6898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:09.456262 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:09.438785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.28.243.62:45281)
0812 16:45:09.439996 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:09.440012 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:09.440074 (+ 62us) server_negotiation.cc:407] Connection header received
0812 16:45:09.442521 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:09.442561 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:09.443254 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:09.443796 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:09.444489 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:09.447149 (+ 2660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:09.449685 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:09.450820 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:09.451093 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:09.452981 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:09.452993 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:09.453017 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:45:09.453880 (+ 863us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:09.454007 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:45:09.454041 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:09.454171 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:09.454439 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 16:45:09.454845 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":351,"threads_started":1}
I20260812 16:45:09.456506 6898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521: Bootstrap starting.
I20260812 16:45:09.460074 6898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:09.460397 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:45:09.461174 6898 log.cc:824] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:09.465749 6898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521: No bootstrap required, opened a new log
I20260812 16:45:09.474609 6898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1884e39f81bd4ac78062ca9ff8407521" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45281 } }
I20260812 16:45:09.475507 6898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:09.475653 6898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1884e39f81bd4ac78062ca9ff8407521, State: Initialized, Role: FOLLOWER
I20260812 16:45:09.476476 6898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1884e39f81bd4ac78062ca9ff8407521" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45281 } }
I20260812 16:45:09.476786 6898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:09.477039 6898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:09.477357 6898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:09.480000 6898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1884e39f81bd4ac78062ca9ff8407521" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45281 } }
I20260812 16:45:09.480654 6898 leader_election.cc:302] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1884e39f81bd4ac78062ca9ff8407521; no voters:
I20260812 16:45:09.481365 6898 leader_election.cc:288] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:09.481524 6904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:09.482343 6904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [term 1 LEADER]: Becoming Leader. State: Replica: 1884e39f81bd4ac78062ca9ff8407521, State: Running, Role: LEADER
I20260812 16:45:09.483055 6904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1884e39f81bd4ac78062ca9ff8407521" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45281 } }
I20260812 16:45:09.483682 6898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:09.487008 6907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1884e39f81bd4ac78062ca9ff8407521. Latest consensus state: current_term: 1 leader_uuid: "1884e39f81bd4ac78062ca9ff8407521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1884e39f81bd4ac78062ca9ff8407521" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45281 } } }
I20260812 16:45:09.486961 6905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1884e39f81bd4ac78062ca9ff8407521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1884e39f81bd4ac78062ca9ff8407521" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45281 } } }
I20260812 16:45:09.487403 6907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:09.487416 6905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:09.494931 6916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:09.500160 6916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:09.509874 6916 catalog_manager.cc:1347] Generated new cluster ID: d958770ed7b04f33961041f77a516b40
I20260812 16:45:09.510020 6916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:09.541589 6916 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:09.543733 6916 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:09.578667 6916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1884e39f81bd4ac78062ca9ff8407521: Generated new TSK 0
I20260812 16:45:09.579705 6916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:45:09.632252 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:09.598683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.28.243.62:45281)
0812 16:45:09.598904 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:09.598941 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:09.599362 (+ 421us) server_negotiation.cc:407] Connection header received
0812 16:45:09.599583 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:09.599589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:09.599716 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:09.599919 (+ 203us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:45:09.600383 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:09.601522 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:09.603009 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:09.603738 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:09.603876 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:09.604812 (+ 936us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:45:09.631464 (+ 26652us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 16:45:09.631889 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 16:45:09.632411 6902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 16:45:09.632457 6952 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:09.598476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:45281 (local address 127.0.0.1:41336)
0812 16:45:09.599251 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:09.599269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:09.599408 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:09.599960 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:09.599963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:09.599974 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:45:09.600236 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:09.600244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:09.601685 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:09.601690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:09.602836 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:09.602848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:09.604087 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:09.604094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:09.604418 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:09.604478 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:45:09.631825 (+ 27347us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 16:45:09.632094 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.62:45281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":462,"threads_started":1}
I20260812 16:45:09.634687 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 6795
I20260812 16:45:09.636622 6893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:09.739472 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 6795
2026-08-12T16:45:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (924 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (478 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (473 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:45:10 dist-test-slave-1vmg krb5kdc[6974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:45:10 dist-test-slave-1vmg krb5kdc[6974](info): set up 2 sockets
Aug 12 16:45:10 dist-test-slave-1vmg krb5kdc[6974](info): commencing operation
krb5kdc: starting...
W20260812 16:45:12.010079 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:45:12 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:45:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:45:12.333244 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:41137
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:12.769603 6990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:12.770128 6990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:12.770277 6990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:12.779287 6990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:12.779459 6990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:12.779548 6990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:12.779634 6990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:12.779700 6990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:12.794345 6990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:12.796638 6990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:12.798152 6990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:12.815802 6999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:12.815765 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:12.816219 7001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:12.818164 6990 server_base.cc:1034] running on GCE node
I20260812 16:45:12.819547 6990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:12.821728 6990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:12.823098 6990 hybrid_clock.cc:648] HybridClock initialized: now 1786553112823029 us; error 56 us; skew 500 ppm
I20260812 16:45:12.823576 6990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:12 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553112, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:12.827167 6990 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:45:12.829177 6990 webserver.cc:460] Webserver started at http://127.28.243.62:39659/ using document root <none> and password file <none>
I20260812 16:45:12.829921 6990 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:12.830031 6990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:12.830314 6990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:12.834553 6990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b"
format_stamp: "Formatted at 2026-08-12 16:45:12 on dist-test-slave-1vmg"
server_key: "3fe9e6a7b1693c4c024796ec2f6da2bb"
server_key_iv: "6263012107316eaceacee545dd59068f"
server_key_version: "encryptionkey@0"
I20260812 16:45:12.835533 6990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b"
format_stamp: "Formatted at 2026-08-12 16:45:12 on dist-test-slave-1vmg"
server_key: "3fe9e6a7b1693c4c024796ec2f6da2bb"
server_key_iv: "6263012107316eaceacee545dd59068f"
server_key_version: "encryptionkey@0"
I20260812 16:45:12.841280 6990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:45:12.845521 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:12.849372 6990 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260812 16:45:12.849707 6990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b"
format_stamp: "Formatted at 2026-08-12 16:45:12 on dist-test-slave-1vmg"
server_key: "3fe9e6a7b1693c4c024796ec2f6da2bb"
server_key_iv: "6263012107316eaceacee545dd59068f"
server_key_version: "encryptionkey@0"
I20260812 16:45:12.850178 6990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:12.873471 6990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:12.879424 6990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:12.879725 6990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:12.903700 6990 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:41137
I20260812 16:45:12.903752 7105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:41137 every 8 connection(s)
I20260812 16:45:12.905159 6990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:12.908581 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 6990
I20260812 16:45:12.908918 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:12.910092 29644 external_mini_cluster.cc:1428] Setting key 15c3cc8d9b431666286dbcc605478891
I20260812 16:45:12.911135 7106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:12.926468 7106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b: Bootstrap starting.
Aug 12 16:45:12 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:12.932253 7106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:12.935130 7106 log.cc:824] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:12.939666 7106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b: No bootstrap required, opened a new log
I20260812 16:45:12.940979 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:12.912706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.28.243.62:41137)
0812 16:45:12.915011 (+ 2305us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:12.915039 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:12.915128 (+ 89us) server_negotiation.cc:407] Connection header received
0812 16:45:12.917047 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:12.917079 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:12.917634 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:12.918088 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:12.919768 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:12.921127 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:12.922439 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:12.923295 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:12.923491 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:12.928590 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:12.928613 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:12.928637 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:12.928704 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:12.935130 (+ 6426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:12.936414 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:12.936426 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:12.936434 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:12.936685 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:12.937224 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:12.937230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:12.937233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:12.938062 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:12.938403 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:12.938870 (+ 467us) server_negotiation.cc:299] Negotiation successful
0812 16:45:12.939438 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":463,"threads_started":1}
I20260812 16:45:12.951611 7106 raft_consensus.cc:348] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41137 } }
I20260812 16:45:12.952138 7106 raft_consensus.cc:374] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:12.952347 7106 raft_consensus.cc:729] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31efdf1ee1d455da32c9caa4bc9cd6b, State: Initialized, Role: FOLLOWER
I20260812 16:45:12.953256 7106 consensus_queue.cc:260] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41137 } }
I20260812 16:45:12.953568 7106 raft_consensus.cc:388] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:12.953691 7106 raft_consensus.cc:482] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:12.953857 7106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:12.957150 7106 raft_consensus.cc:504] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41137 } }
I20260812 16:45:12.958161 7106 leader_election.cc:302] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f31efdf1ee1d455da32c9caa4bc9cd6b; no voters:
I20260812 16:45:12.958842 7106 leader_election.cc:288] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:12.959046 7112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:12.959741 7112 raft_consensus.cc:686] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [term 1 LEADER]: Becoming Leader. State: Replica: f31efdf1ee1d455da32c9caa4bc9cd6b, State: Running, Role: LEADER
I20260812 16:45:12.960412 7112 consensus_queue.cc:237] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41137 } }
I20260812 16:45:12.961200 7106 sys_catalog.cc:564] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:12.963349 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41137 } } }
I20260812 16:45:12.964180 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:12.963552 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f31efdf1ee1d455da32c9caa4bc9cd6b. Latest consensus state: current_term: 1 leader_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31efdf1ee1d455da32c9caa4bc9cd6b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41137 } } }
I20260812 16:45:12.965993 7129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:12.968096 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:12.977715 7129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:12.991297 7129 catalog_manager.cc:1347] Generated new cluster ID: a21c31e6ed9b4b928882add85bbeb9fa
I20260812 16:45:12.991578 7129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:13.007025 7129 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:13.009088 7129 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:13.025496 7129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f31efdf1ee1d455da32c9caa4bc9cd6b: Generated new TSK 0
I20260812 16:45:13.027762 7129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:45:13.103098 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41137
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:13.485577 7150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:13.485906 7150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:13.486042 7150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:13.493727 7150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:13.493884 7150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:13.494045 7150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:13.504191 7150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:13.505658 7150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:13.507124 7150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:13.521019 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:13.522083 7160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:13.523772 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:13.524713 7150 server_base.cc:1034] running on GCE node
I20260812 16:45:13.525651 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:13.527032 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:13.528326 7150 hybrid_clock.cc:648] HybridClock initialized: now 1786553113528273 us; error 57 us; skew 500 ppm
I20260812 16:45:13.528699 7150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:13 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:13.532661 7150 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:45:13.534591 7150 webserver.cc:460] Webserver started at http://127.28.243.1:36423/ using document root <none> and password file <none>
I20260812 16:45:13.535449 7150 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:13.535581 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:13.535913 7150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:13.539254 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "b5ae7c3e774e49939c2cec18bd4e244d"
format_stamp: "Formatted at 2026-08-12 16:45:13 on dist-test-slave-1vmg"
server_key: "f624077d8868f31f72174012bfca7a13"
server_key_iv: "4e6a7bbadb9c39654ac04c5d30fba302"
server_key_version: "encryptionkey@0"
I20260812 16:45:13.540122 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "b5ae7c3e774e49939c2cec18bd4e244d"
format_stamp: "Formatted at 2026-08-12 16:45:13 on dist-test-slave-1vmg"
server_key: "f624077d8868f31f72174012bfca7a13"
server_key_iv: "4e6a7bbadb9c39654ac04c5d30fba302"
server_key_version: "encryptionkey@0"
I20260812 16:45:13.545840 7150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:45:13.549829 7175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:13.551456 7150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:45:13.551645 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "b5ae7c3e774e49939c2cec18bd4e244d"
format_stamp: "Formatted at 2026-08-12 16:45:13 on dist-test-slave-1vmg"
server_key: "f624077d8868f31f72174012bfca7a13"
server_key_iv: "4e6a7bbadb9c39654ac04c5d30fba302"
server_key_version: "encryptionkey@0"
I20260812 16:45:13.551864 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:13.577908 7150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:13.583696 7150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:13.583977 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:13.585036 7150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:13.586547 7150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:13.586710 7150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:13.586855 7150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:13.586902 7150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:13.621217 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:44843 every 8 connection(s)
I20260812 16:45:13.621234 7150 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:44843
I20260812 16:45:13.623265 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:13.625550 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 7150
I20260812 16:45:13.625778 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:45:13.626446 29644 external_mini_cluster.cc:1428] Setting key dc0e2d57a242d935583d6a3895e05039
Aug 12 16:45:13 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:13.648478 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:13.627472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:48207 (local address 127.28.243.62:41137)
0812 16:45:13.627993 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:13.627997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:13.629264 (+ 1267us) server_negotiation.cc:407] Connection header received
0812 16:45:13.631261 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:13.631268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:13.631357 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:13.631484 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:13.633223 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:13.634229 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:13.636428 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:13.636999 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:13.637136 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:13.641686 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:13.641693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:13.641697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:13.641740 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:13.644416 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:13.645623 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:13.645628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:13.645631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:13.645846 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:13.646475 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:13.646479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:13.646481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:13.646790 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:13.646950 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:13.647989 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0812 16:45:13.648204 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":270,"threads_started":1}
I20260812 16:45:13.649951 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:13.628127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:41137 (local address 127.28.243.1:48207)
0812 16:45:13.629071 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:13.629119 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:13.630974 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:13.631760 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:13.631770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:13.632374 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:13.633014 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:13.633036 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:13.634437 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:13.634442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:13.636245 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:13.636264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:13.637256 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:13.637262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:13.637542 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:13.638491 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:13.638521 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:13.641369 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:13.644635 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:13.644646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:13.644667 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:13.645425 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:13.646001 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:13.646007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:13.646010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:13.646311 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:13.646989 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:13.646998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:13.647499 (+ 501us) client_negotiation.cc:769] Sending connection context
0812 16:45:13.647829 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 16:45:13.648419 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":363,"threads_started":1}
I20260812 16:45:13.651865 7374 heartbeater.cc:347] Connected to a master server at 127.28.243.62:41137
I20260812 16:45:13.652206 7374 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:13.652998 7374 heartbeater.cc:511] Master 127.28.243.62:41137 requested a full tablet report, sending...
I20260812 16:45:13.655309 7052 ts_manager.cc:194] Registered new tserver with Master: b5ae7c3e774e49939c2cec18bd4e244d (127.28.243.1:44843)
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:45:13.657059 7052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:48207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:45:13.695292 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41137
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:14.088431 7383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:14.088779 7383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:14.088881 7383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:14.096599 7383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:14.097071 7383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:14.097414 7383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:14.108057 7383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:14.109519 7383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:14.110999 7383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:14.125716 7391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:14.125784 7396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:14.126463 7388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:14.126721 7383 server_base.cc:1034] running on GCE node
I20260812 16:45:14.127575 7383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:14.128660 7383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:14.129928 7383 hybrid_clock.cc:648] HybridClock initialized: now 1786553114129866 us; error 54 us; skew 500 ppm
I20260812 16:45:14.130299 7383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.134122 7383 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:45:14.135926 7383 webserver.cc:460] Webserver started at http://127.28.243.2:33083/ using document root <none> and password file <none>
I20260812 16:45:14.136694 7383 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:14.136830 7383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:14.137168 7383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:14.140676 7383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "70a49fedda824ec193676b1d5be123bd"
format_stamp: "Formatted at 2026-08-12 16:45:14 on dist-test-slave-1vmg"
server_key: "b4443621594f31437c0f4d3756d1497b"
server_key_iv: "341e0edb04729253b289e2375983a6ed"
server_key_version: "encryptionkey@0"
I20260812 16:45:14.141623 7383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "70a49fedda824ec193676b1d5be123bd"
format_stamp: "Formatted at 2026-08-12 16:45:14 on dist-test-slave-1vmg"
server_key: "b4443621594f31437c0f4d3756d1497b"
server_key_iv: "341e0edb04729253b289e2375983a6ed"
server_key_version: "encryptionkey@0"
I20260812 16:45:14.148290 7383 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:45:14.153685 7404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:14.155821 7383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:45:14.156047 7383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "70a49fedda824ec193676b1d5be123bd"
format_stamp: "Formatted at 2026-08-12 16:45:14 on dist-test-slave-1vmg"
server_key: "b4443621594f31437c0f4d3756d1497b"
server_key_iv: "341e0edb04729253b289e2375983a6ed"
server_key_version: "encryptionkey@0"
I20260812 16:45:14.156306 7383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:14.175100 7383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:14.180534 7383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:14.180886 7383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:14.182049 7383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:14.183696 7383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:14.183794 7383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:14.183940 7383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:14.184006 7383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:14.224790 7383 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:34775
I20260812 16:45:14.224793 7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:34775 every 8 connection(s)
I20260812 16:45:14.226279 7383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:14.236733 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 7383
I20260812 16:45:14.236974 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:45:14.237514 29644 external_mini_cluster.cc:1428] Setting key 9e6e1c0b73651b695625671d7cfb6351
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:14.257025 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:14.230411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:42361 (local address 127.28.243.62:41137)
0812 16:45:14.231201 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:14.231206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:14.233360 (+ 2154us) server_negotiation.cc:407] Connection header received
0812 16:45:14.235921 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:14.235929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:14.236030 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:14.236143 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:14.238083 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.238828 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.241602 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.242071 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.242197 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:14.247378 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:14.247393 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:14.247397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:14.247444 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:14.252716 (+ 5272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:14.254010 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:14.254018 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:14.254021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:14.254243 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:14.254783 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:14.254787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:14.254790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:14.255309 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:14.255487 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:14.256348 (+ 861us) server_negotiation.cc:299] Negotiation successful
0812 16:45:14.256628 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":307,"threads_started":1}
I20260812 16:45:14.258108 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:14.230957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:41137 (local address 127.28.243.2:42361)
0812 16:45:14.233026 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:14.233126 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:14.235512 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:14.236448 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:14.236459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:14.237154 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:14.237863 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:14.237887 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.239236 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.239242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:14.241392 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:14.241412 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.242362 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.242369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:14.242689 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:14.243535 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:14.243568 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:14.246764 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:14.253221 (+ 6457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:14.253231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:14.253250 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:14.253822 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:14.254402 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:14.254407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:14.254409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:14.254648 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:14.255458 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:14.255465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:14.255853 (+ 388us) client_negotiation.cc:769] Sending connection context
0812 16:45:14.256132 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 16:45:14.256806 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":823,"threads_started":1}
I20260812 16:45:14.259712 7604 heartbeater.cc:347] Connected to a master server at 127.28.243.62:41137
I20260812 16:45:14.260141 7604 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:14.261111 7604 heartbeater.cc:511] Master 127.28.243.62:41137 requested a full tablet report, sending...
I20260812 16:45:14.263149 7052 ts_manager.cc:194] Registered new tserver with Master: 70a49fedda824ec193676b1d5be123bd (127.28.243.2:34775)
I20260812 16:45:14.264058 7052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:42361
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:45:14.307683 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41137
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:14.661176 7374 heartbeater.cc:503] Master 127.28.243.62:41137 was elected leader, sending a full tablet report...
W20260812 16:45:14.724660 7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:14.724999 7618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:14.725111 7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:14.735141 7618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:14.735327 7618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:14.735481 7618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:14.749971 7618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:14.752337 7618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:14.754056 7618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 16:45:14.772038 7618 server_base.cc:1034] running on GCE node
W20260812 16:45:14.771925 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:14.771925 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:14.772089 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:14.773442 7618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:14.774672 7618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:14.776026 7618 hybrid_clock.cc:648] HybridClock initialized: now 1786553114775940 us; error 74 us; skew 500 ppm
I20260812 16:45:14.776641 7618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.780418 7618 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:45:14.782228 7618 webserver.cc:460] Webserver started at http://127.28.243.3:46357/ using document root <none> and password file <none>
I20260812 16:45:14.783077 7618 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:14.783222 7618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:14.783568 7618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:14.786885 7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "71906ef2ba734b4fa5083e24cc92bc18"
format_stamp: "Formatted at 2026-08-12 16:45:14 on dist-test-slave-1vmg"
server_key: "c9cc6f1b038bc22761e0c2d777040bdc"
server_key_iv: "bdf3cdc3d386633cb243fb7cbc721046"
server_key_version: "encryptionkey@0"
I20260812 16:45:14.787988 7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "71906ef2ba734b4fa5083e24cc92bc18"
format_stamp: "Formatted at 2026-08-12 16:45:14 on dist-test-slave-1vmg"
server_key: "c9cc6f1b038bc22761e0c2d777040bdc"
server_key_iv: "bdf3cdc3d386633cb243fb7cbc721046"
server_key_version: "encryptionkey@0"
I20260812 16:45:14.794745 7618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:45:14.799101 7653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:14.800762 7618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:14.800959 7618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "71906ef2ba734b4fa5083e24cc92bc18"
format_stamp: "Formatted at 2026-08-12 16:45:14 on dist-test-slave-1vmg"
server_key: "c9cc6f1b038bc22761e0c2d777040bdc"
server_key_iv: "bdf3cdc3d386633cb243fb7cbc721046"
server_key_version: "encryptionkey@0"
I20260812 16:45:14.801203 7618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:14.824515 7618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:14.830286 7618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:14.830601 7618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:14.831722 7618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:14.833362 7618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:14.833454 7618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:14.833590 7618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:14.833647 7618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:14.869434 7618 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:33195
I20260812 16:45:14.869511 7840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:33195 every 8 connection(s)
I20260812 16:45:14.870939 7618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:14.881040 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 7618
I20260812 16:45:14.881217 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:45:14.881676 29644 external_mini_cluster.cc:1428] Setting key e3e6453129a1e80d4bcae8fd5d2e21f6
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:14.899214 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:14.874968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:42873 (local address 127.28.243.62:41137)
0812 16:45:14.875769 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:14.875775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:14.877026 (+ 1251us) server_negotiation.cc:407] Connection header received
0812 16:45:14.879281 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:14.879287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:14.879408 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:14.879516 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:14.881540 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.882390 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.884455 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.884999 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.885151 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:14.890191 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:14.890198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:14.890202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:14.890250 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:14.893745 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:14.895455 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:14.895461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:14.895463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:14.895665 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:14.896656 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:14.896663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:14.896666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:14.897115 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:14.897318 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:14.898557 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0812 16:45:14.898861 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":374,"threads_started":1}
I20260812 16:45:14.900211 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:14.875635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:41137 (local address 127.28.243.3:42873)
0812 16:45:14.876761 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:14.876836 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:14.878823 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:14.879874 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:14.879885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:14.880523 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:14.881332 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:14.881356 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.882569 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.882574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:14.884280 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:14.884299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.885311 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.885317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:14.885558 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:14.886386 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:14.886417 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:14.889700 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:14.894044 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:14.894055 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:14.894074 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:14.895092 (+ 1018us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:14.895863 (+ 771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:14.895868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:14.895871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:14.896247 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:14.897395 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:14.897407 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:14.897982 (+ 575us) client_negotiation.cc:769] Sending connection context
0812 16:45:14.898346 (+ 364us) client_negotiation.cc:240] Negotiation successful
0812 16:45:14.898980 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":423,"threads_started":1}
I20260812 16:45:14.902474 7843 heartbeater.cc:347] Connected to a master server at 127.28.243.62:41137
I20260812 16:45:14.902860 7843 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:14.903942 7843 heartbeater.cc:511] Master 127.28.243.62:41137 requested a full tablet report, sending...
I20260812 16:45:14.905946 7052 ts_manager.cc:194] Registered new tserver with Master: 71906ef2ba734b4fa5083e24cc92bc18 (127.28.243.3:33195)
I20260812 16:45:14.906947 7052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:42873
I20260812 16:45:14.912636 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/test-admin.keytab.
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.950996 29644 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 16:45:14.958957 29644 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc' of type FILE
W20260812 16:45:14.959152 7857 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:14.955506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.28.243.1:44843)
0812 16:45:14.956287 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:14.956296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:14.956527 (+ 231us) server_negotiation.cc:407] Connection header received
0812 16:45:14.956657 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:14.956666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:14.956933 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:14.957100 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:14.958486 (+ 1386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40514: BlockingRecv error: recv got EOF from 127.0.0.1:40514 (error 108)
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":426,"threads_started":1}
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.961205 29644 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.962951 29644 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:45:14.983462 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:14.965170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.28.243.1:44843)
0812 16:45:14.965438 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:14.965443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:14.965591 (+ 148us) server_negotiation.cc:407] Connection header received
0812 16:45:14.965717 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:14.965723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:14.965790 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:14.965885 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:14.967575 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.968909 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.970864 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:14.971569 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:14.971732 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:14.976631 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:14.976649 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:14.976667 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:14.976715 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:14.980125 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:14.981106 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:14.981115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:14.981117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:14.981277 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:14.981691 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:14.981695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:14.981697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:14.982129 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:14.982437 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:14.982806 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 16:45:14.983131 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 16:45:14.990684 7857 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:14.988729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.28.243.1:44843)
0812 16:45:14.988982 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:14.988993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:14.989148 (+ 155us) server_negotiation.cc:407] Connection header received
0812 16:45:14.989355 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:14.989359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:14.989427 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:14.989517 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:14.990397 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40524: BlockingRecv error: recv got EOF from 127.0.0.1:40524 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260812 16:45:14.991545 29644 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc)
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.993882 29644 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 16:45:14 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:14.995733 29644 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:45:15.015677 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:14.997413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.28.243.1:44843)
0812 16:45:14.997574 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:14.997578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:14.998168 (+ 590us) server_negotiation.cc:407] Connection header received
0812 16:45:14.998373 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:14.998377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:14.998445 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:14.998528 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:14.999669 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.000395 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.001640 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.002094 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.002218 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:15.007149 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:15.007156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:15.007160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:15.007207 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:15.011053 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.012481 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.012490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.012494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.012868 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.013543 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.013552 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.013554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.013968 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:15.014260 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:15.014891 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 16:45:15.015344 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 16:45:15.020991 7857 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:15.019071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.28.243.1:44843)
0812 16:45:15.019305 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:15.019312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:15.019606 (+ 294us) server_negotiation.cc:407] Connection header received
0812 16:45:15.019793 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:15.019798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:15.019877 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:15.019965 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:15.020768 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40544: BlockingRecv error: recv got EOF from 127.0.0.1:40544 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 16:45:15.021189 29644 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786553056142401-29644-0/krb5kdc/krb5cc' of type FILE
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:15.022974 29644 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[6974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:15.024442 29644 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:45:15.041042 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:15.026144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.28.243.1:44843)
0812 16:45:15.026397 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:15.026404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:15.026739 (+ 335us) server_negotiation.cc:407] Connection header received
0812 16:45:15.026862 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:15.026867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:15.027003 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:15.027090 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:15.028119 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.029395 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.030898 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.031421 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.031534 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:15.035472 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:15.035479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:15.035482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:15.035523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:15.038226 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.039104 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.039109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.039112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.039269 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.039654 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.039657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.039659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.039949 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:15.040197 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:15.040481 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 16:45:15.040685 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 16:45:15.042995 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 7150
I20260812 16:45:15.059151 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:15.043857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.28.243.1:44843)
0812 16:45:15.044179 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:15.044186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:15.044482 (+ 296us) server_negotiation.cc:407] Connection header received
0812 16:45:15.044654 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:15.044659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:15.044724 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:15.044812 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:15.045887 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.046581 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.047881 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.048489 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.048630 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:15.052483 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:15.052489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:15.052492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:15.052538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:15.055445 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.056287 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.056293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.056295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.056443 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.056889 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.056894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.056897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.057268 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:15.057526 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:15.058111 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 16:45:15.058645 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 16:45:15.059943 7368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:15.150254 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 7150
I20260812 16:45:15.173372 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 7383
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:45:15.196393 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:15.175190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.28.243.2:34775)
0812 16:45:15.175858 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:15.175866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:15.175896 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:45:15.175986 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:15.175996 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:15.176232 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:15.176393 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:15.177750 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.179383 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.181189 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.182064 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.182217 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:15.187476 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:15.187496 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:15.187514 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:15.187562 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:15.191118 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.192654 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.192664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.192666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.192972 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.193916 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.193923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.193926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.194586 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:15.195039 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:15.195746 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 16:45:15.196034 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":349,"threads_started":1}
I20260812 16:45:15.197978 7597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:15.292536 7604 heartbeater.cc:503] Master 127.28.243.62:41137 was elected leader, sending a full tablet report...
I20260812 16:45:15.295050 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 7383
I20260812 16:45:15.322506 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 7618
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[6974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:45:15.344501 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:15.323671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.28.243.3:33195)
0812 16:45:15.324237 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:15.324245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:15.324276 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:45:15.324347 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:15.324354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:15.324566 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:15.324709 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:15.326327 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.328130 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.330463 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:15.331694 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:15.331889 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:15.337234 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:15.337257 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:15.337275 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:15.337336 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:15.341042 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.341967 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.341976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.341978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.342149 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:15.342592 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:15.342596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:15.342598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:15.343120 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:15.343458 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:15.343804 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 16:45:15.344141 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":254,"threads_started":1}
I20260812 16:45:15.345870 7831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:15.443511 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 7618
I20260812 16:45:15.474689 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 6990
I20260812 16:45:15.476393 7098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:15.602839 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 6990
2026-08-12T16:45:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4925 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[7897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[7897](info): set up 2 sockets
Aug 12 16:45:15 dist-test-slave-1vmg krb5kdc[7897](info): commencing operation
krb5kdc: starting...
W20260812 16:45:16.980046 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.009s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:45:17 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab.
Entry for principal oryx/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab.
I20260812 16:45:17.304884 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46243
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab
--principal=oryx/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:17.763478 7915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:17.763849 7915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:17.763988 7915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:17.773947 7915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:17.774389 7915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:17.775442 7915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:17.775736 7915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:17.775866 7915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:17.788677 7915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab
--principal=oryx/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:17.790768 7915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:17.793166 7915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:17.809316 7929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:17.809544 7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:17.809492 7934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:17.810710 7915 server_base.cc:1034] running on GCE node
I20260812 16:45:17.812085 7915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:17.814355 7915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:17.815919 7915 hybrid_clock.cc:648] HybridClock initialized: now 1786553117815800 us; error 108 us; skew 500 ppm
I20260812 16:45:17.816377 7915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:17 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:17.823462 7915 init.cc:377] Logged in from keytab as oryx/127.28.243.62@KRBTEST.COM (short username oryx)
I20260812 16:45:17.825367 7915 webserver.cc:460] Webserver started at http://127.28.243.62:45491/ using document root <none> and password file <none>
I20260812 16:45:17.826184 7915 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:17.826529 7915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:17.827095 7915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:17.832721 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "20fa166f837c466a9f213e8d8c7d2cfd"
format_stamp: "Formatted at 2026-08-12 16:45:17 on dist-test-slave-1vmg"
server_key: "75bd74b6c08843cbfeb2b8ac83a3df70"
server_key_iv: "2cae225fb91605de93b311e0f1d433f0"
server_key_version: "encryptionkey@0"
I20260812 16:45:17.834342 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "20fa166f837c466a9f213e8d8c7d2cfd"
format_stamp: "Formatted at 2026-08-12 16:45:17 on dist-test-slave-1vmg"
server_key: "75bd74b6c08843cbfeb2b8ac83a3df70"
server_key_iv: "2cae225fb91605de93b311e0f1d433f0"
server_key_version: "encryptionkey@0"
I20260812 16:45:17.840953 7915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:45:17.845039 7946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:17.846905 7915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:45:17.847342 7915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "20fa166f837c466a9f213e8d8c7d2cfd"
format_stamp: "Formatted at 2026-08-12 16:45:17 on dist-test-slave-1vmg"
server_key: "75bd74b6c08843cbfeb2b8ac83a3df70"
server_key_iv: "2cae225fb91605de93b311e0f1d433f0"
server_key_version: "encryptionkey@0"
I20260812 16:45:17.847576 7915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:17.871544 7915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:17.878074 7915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:17.878449 7915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:17.899824 7915 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:46243
I20260812 16:45:17.899878 8027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:46243 every 8 connection(s)
I20260812 16:45:17.901422 7915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:17.903795 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 7915
I20260812 16:45:17.903995 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:17.904659 29644 external_mini_cluster.cc:1428] Setting key 5f975e9ceaa269e1d4989286a989f55a
I20260812 16:45:17.907096 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:17.921327 8031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Bootstrap starting.
Aug 12 16:45:17 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:17.924826 8031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:17.926110 8031 log.cc:824] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:17.930701 8031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: No bootstrap required, opened a new log
I20260812 16:45:17.936156 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:17.907295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.28.243.62:46243)
0812 16:45:17.908517 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:17.908531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:17.908584 (+ 53us) server_negotiation.cc:407] Connection header received
0812 16:45:17.910770 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:17.910808 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:17.911511 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:17.912378 (+ 867us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:17.914689 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:17.916561 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:17.918398 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:17.919310 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:17.919526 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:17.923852 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:17.923863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:17.923880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:17.923934 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:17.929042 (+ 5108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:17.930211 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:17.930221 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:17.930230 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:17.930482 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:17.931011 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:17.931016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:17.931019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:17.932036 (+ 1017us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:17.932611 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:17.933530 (+ 919us) server_negotiation.cc:299] Negotiation successful
0812 16:45:17.934411 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":459,"threads_started":1}
I20260812 16:45:17.940429 8031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } }
I20260812 16:45:17.940850 8031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:17.940943 8031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fa166f837c466a9f213e8d8c7d2cfd, State: Initialized, Role: FOLLOWER
I20260812 16:45:17.941532 8031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } }
I20260812 16:45:17.941769 8031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:17.941897 8031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:17.942057 8031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:17.944443 8031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } }
I20260812 16:45:17.944994 8031 leader_election.cc:302] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20fa166f837c466a9f213e8d8c7d2cfd; no voters:
I20260812 16:45:17.945688 8031 leader_election.cc:288] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:17.945807 8039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:17.947072 8039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 LEADER]: Becoming Leader. State: Replica: 20fa166f837c466a9f213e8d8c7d2cfd, State: Running, Role: LEADER
I20260812 16:45:17.947669 8039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } }
I20260812 16:45:17.948386 8031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:17.951710 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } }
I20260812 16:45:17.952136 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:17.951922 8043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20fa166f837c466a9f213e8d8c7d2cfd. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } }
I20260812 16:45:17.952292 8043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:17.955085 8048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:17.960381 8048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:17.971941 8048 catalog_manager.cc:1347] Generated new cluster ID: 56169dabf64d41baa83fea0e677ec7f5
I20260812 16:45:17.972151 8048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:17.984997 8048 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:17.986584 8048 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:17.998412 8048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Generated new TSK 0
I20260812 16:45:17.999513 8048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab.
Entry for principal oryx/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab.
I20260812 16:45:18.090160 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46243
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:18.521672 8079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:18.522022 8079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:18.522158 8079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:18.531278 8079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:18.531450 8079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:18.531613 8079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:18.543517 8079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:18.545658 8079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:18.547452 8079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:18.562776 8087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:18.562856 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:18.562837 8093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:18.563808 8079 server_base.cc:1034] running on GCE node
I20260812 16:45:18.564493 8079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:18.565711 8079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:18.567068 8079 hybrid_clock.cc:648] HybridClock initialized: now 1786553118566983 us; error 73 us; skew 500 ppm
I20260812 16:45:18.567485 8079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:18 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553118, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:18.572155 8079 init.cc:377] Logged in from keytab as oryx/127.28.243.1@KRBTEST.COM (short username oryx)
I20260812 16:45:18.574368 8079 webserver.cc:460] Webserver started at http://127.28.243.1:35711/ using document root <none> and password file <none>
I20260812 16:45:18.575557 8079 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:18.575817 8079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:18.576259 8079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:18.579819 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "3a70ef06ada842c69ce906c08766ddcc"
format_stamp: "Formatted at 2026-08-12 16:45:18 on dist-test-slave-1vmg"
server_key: "90a142a8a566309628c2045c9d5c29f6"
server_key_iv: "6f4809421c1dae504220889bf16e3b2a"
server_key_version: "encryptionkey@0"
I20260812 16:45:18.580673 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "3a70ef06ada842c69ce906c08766ddcc"
format_stamp: "Formatted at 2026-08-12 16:45:18 on dist-test-slave-1vmg"
server_key: "90a142a8a566309628c2045c9d5c29f6"
server_key_iv: "6f4809421c1dae504220889bf16e3b2a"
server_key_version: "encryptionkey@0"
I20260812 16:45:18.587203 8079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 16:45:18.591503 8107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:18.593271 8079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:45:18.593464 8079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "3a70ef06ada842c69ce906c08766ddcc"
format_stamp: "Formatted at 2026-08-12 16:45:18 on dist-test-slave-1vmg"
server_key: "90a142a8a566309628c2045c9d5c29f6"
server_key_iv: "6f4809421c1dae504220889bf16e3b2a"
server_key_version: "encryptionkey@0"
I20260812 16:45:18.593690 8079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:18.612089 8079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:18.617069 8079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:18.617404 8079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:18.618669 8079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:18.620550 8079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:18.620649 8079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:18.620776 8079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:18.620832 8079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:18.658264 8079 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:34481
I20260812 16:45:18.658274 8299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:34481 every 8 connection(s)
I20260812 16:45:18.659806 8079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:18.668552 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 8079
I20260812 16:45:18.668817 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:45:18.669801 29644 external_mini_cluster.cc:1428] Setting key ba8b68828f4c1abc02e82e76b77603dc
Aug 12 16:45:18 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553118, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:18.688730 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:18.664038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:55673 (local address 127.28.243.62:46243)
0812 16:45:18.664544 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:18.664549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:18.666077 (+ 1528us) server_negotiation.cc:407] Connection header received
0812 16:45:18.668299 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:18.668305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:18.668412 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:18.668517 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:18.670473 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:18.671524 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:18.674069 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:18.674499 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:18.674622 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:18.680172 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:18.680179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:18.680183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:18.680242 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:18.683089 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:18.684664 (+ 1575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:18.684672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:18.684674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:18.684910 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:18.685674 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:18.685681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:18.685684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:18.686114 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:18.686300 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:18.687691 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0812 16:45:18.688287 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260812 16:45:18.689284 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:18.664666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:55673)
0812 16:45:18.665762 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:18.665836 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:18.667912 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:18.668865 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:18.668878 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:18.669528 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:18.670247 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:18.670272 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:18.671799 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:18.671804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:18.673799 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:18.673823 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:18.674807 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:18.674814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:18.675191 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:18.676062 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:18.676096 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:18.679555 (+ 3459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:18.683391 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:18.683406 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:18.683438 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:18.684383 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:18.685069 (+ 686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:18.685074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:18.685077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:18.685411 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:18.686319 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:18.686330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:18.687110 (+ 780us) client_negotiation.cc:769] Sending connection context
0812 16:45:18.687447 (+ 337us) client_negotiation.cc:240] Negotiation successful
0812 16:45:18.687971 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":406,"threads_started":1}
I20260812 16:45:18.691273 8301 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46243
I20260812 16:45:18.691674 8301 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:18.692718 8301 heartbeater.cc:511] Master 127.28.243.62:46243 requested a full tablet report, sending...
I20260812 16:45:18.695880 7971 ts_manager.cc:194] Registered new tserver with Master: 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481)
I20260812 16:45:18.697921 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:55673
WARNING: no policy specified for oryx/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab.
Entry for principal oryx/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab.
I20260812 16:45:18.745358 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46243
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:19.148964 8314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:19.149329 8314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:19.149477 8314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:19.157624 8314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:19.157796 8314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:19.157986 8314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:19.168862 8314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:19.170300 8314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:19.171797 8314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:19.187633 8322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:19.187595 8323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:19.187685 8327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:19.188850 8314 server_base.cc:1034] running on GCE node
I20260812 16:45:19.189577 8314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:19.190603 8314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:19.191812 8314 hybrid_clock.cc:648] HybridClock initialized: now 1786553119191760 us; error 44 us; skew 500 ppm
I20260812 16:45:19.192160 8314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:19 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553119, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:19.195724 8314 init.cc:377] Logged in from keytab as oryx/127.28.243.2@KRBTEST.COM (short username oryx)
I20260812 16:45:19.197492 8314 webserver.cc:460] Webserver started at http://127.28.243.2:45641/ using document root <none> and password file <none>
I20260812 16:45:19.198251 8314 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:19.198365 8314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:19.198666 8314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:19.202039 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "34b21184ad5e45e6bc9142e21ce3a705"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "8cf6e5bf144e4fbc8a444d619684e648"
server_key_iv: "fffd8817ea123c6af846522223ac0d5f"
server_key_version: "encryptionkey@0"
I20260812 16:45:19.202894 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "34b21184ad5e45e6bc9142e21ce3a705"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "8cf6e5bf144e4fbc8a444d619684e648"
server_key_iv: "fffd8817ea123c6af846522223ac0d5f"
server_key_version: "encryptionkey@0"
I20260812 16:45:19.210150 8314 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 16:45:19.215523 8339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:19.217787 8314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:45:19.218098 8314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "34b21184ad5e45e6bc9142e21ce3a705"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "8cf6e5bf144e4fbc8a444d619684e648"
server_key_iv: "fffd8817ea123c6af846522223ac0d5f"
server_key_version: "encryptionkey@0"
I20260812 16:45:19.218372 8314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:19.238997 8314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:19.244634 8314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:19.245096 8314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:19.246688 8314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:19.248864 8314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:19.248976 8314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:19.249125 8314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:19.249186 8314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:19.289147 8314 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:39297
I20260812 16:45:19.289265 8534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:39297 every 8 connection(s)
I20260812 16:45:19.291164 8314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:19.300190 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 8314
I20260812 16:45:19.300518 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:45:19.301466 29644 external_mini_cluster.cc:1428] Setting key a6dccf953e646596a06e674bbcaecc62
Aug 12 16:45:19 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553119, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.243.3@KRBTEST.COM; defaulting to no policy
I20260812 16:45:19.337941 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:19.295213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:39895 (local address 127.28.243.62:46243)
0812 16:45:19.297315 (+ 2102us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:19.297320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:19.303098 (+ 5778us) server_negotiation.cc:407] Connection header received
0812 16:45:19.307180 (+ 4082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:19.307187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:19.307306 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:19.307453 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:19.310282 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:19.311850 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:19.314424 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:19.315194 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:19.315372 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:19.320780 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:19.320788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:19.320793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:19.320847 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:19.333320 (+ 12473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:19.334610 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:19.334618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:19.334622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:19.334819 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:19.335444 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:19.335449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:19.335452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:19.335874 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:19.336032 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:19.337292 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0812 16:45:19.337548 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1972,"thread_start_us":1819,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.3@KRBTEST.COM" created.
I20260812 16:45:19.339211 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:19.295789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:39895)
0812 16:45:19.301016 (+ 5227us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:19.301156 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:19.304429 (+ 3273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:19.308049 (+ 3620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:19.308065 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:19.308950 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:19.309897 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:19.309933 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:19.312350 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:19.312356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:19.314133 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:19.314164 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:19.315639 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:19.315650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:19.316085 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:19.317016 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:19.317055 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:19.320380 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:19.333631 (+ 13251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:19.333646 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:19.333671 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:19.334435 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:19.334981 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:19.334986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:19.334990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:19.335304 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:19.336015 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:19.336026 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:19.336497 (+ 471us) client_negotiation.cc:769] Sending connection context
0812 16:45:19.336844 (+ 347us) client_negotiation.cc:240] Negotiation successful
0812 16:45:19.337524 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4558,"thread_start_us":5561,"threads_started":1}
I20260812 16:45:19.341264 8535 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46243
I20260812 16:45:19.341655 8535 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:19.342712 8535 heartbeater.cc:511] Master 127.28.243.62:46243 requested a full tablet report, sending...
I20260812 16:45:19.344864 7971 ts_manager.cc:194] Registered new tserver with Master: 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:19.346189 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:39895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab.
Entry for principal oryx/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab.
I20260812 16:45:19.384222 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46243
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:19.702304 8301 heartbeater.cc:503] Master 127.28.243.62:46243 was elected leader, sending a full tablet report...
W20260812 16:45:19.816133 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:19.816556 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:19.816699 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:19.825968 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:19.826134 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:19.826305 8547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:19.839587 8547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:19.841166 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:19.842687 8547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:19.859679 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:19.859653 8554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:19.859688 8553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:19.861078 8547 server_base.cc:1034] running on GCE node
I20260812 16:45:19.862141 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:19.863739 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:19.865170 8547 hybrid_clock.cc:648] HybridClock initialized: now 1786553119865074 us; error 77 us; skew 500 ppm
I20260812 16:45:19.865629 8547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:19 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553119, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:19.869504 8547 init.cc:377] Logged in from keytab as oryx/127.28.243.3@KRBTEST.COM (short username oryx)
I20260812 16:45:19.871901 8547 webserver.cc:460] Webserver started at http://127.28.243.3:33737/ using document root <none> and password file <none>
I20260812 16:45:19.872725 8547 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:19.872956 8547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:19.873349 8547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:19.877858 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "36ea8cbefec94008a079074cdee0247f"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "6575aade403d4895f532edde8184da6d"
server_key_iv: "440bae3a7a15aad41a383c5ae73ef597"
server_key_version: "encryptionkey@0"
I20260812 16:45:19.879261 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "36ea8cbefec94008a079074cdee0247f"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "6575aade403d4895f532edde8184da6d"
server_key_iv: "440bae3a7a15aad41a383c5ae73ef597"
server_key_version: "encryptionkey@0"
I20260812 16:45:19.887650 8547 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 16:45:19.892697 8575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:19.895236 8547 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:45:19.895493 8547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "36ea8cbefec94008a079074cdee0247f"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "6575aade403d4895f532edde8184da6d"
server_key_iv: "440bae3a7a15aad41a383c5ae73ef597"
server_key_version: "encryptionkey@0"
I20260812 16:45:19.895750 8547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:19.935657 8547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:19.942332 8547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:19.942884 8547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:19.944720 8547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:19.946635 8547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:19.946741 8547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:19.946867 8547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:19.946960 8547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:19.986639 8547 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:37727
I20260812 16:45:19.986683 8773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:37727 every 8 connection(s)
I20260812 16:45:19.988086 8547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:19.993140 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 8547
I20260812 16:45:19.993602 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:45:19.994619 29644 external_mini_cluster.cc:1428] Setting key 4f5f80f46a1762bfdf18c7f4abaef047
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553119, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:20.015923 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:19.992107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:38237 (local address 127.28.243.62:46243)
0812 16:45:19.992661 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:19.992666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:19.994335 (+ 1669us) server_negotiation.cc:407] Connection header received
0812 16:45:19.996729 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:19.996735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:19.996850 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:19.996945 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:19.999057 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:19.999894 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.002452 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.003359 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.003548 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.008427 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.008433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.008436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.008481 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.012297 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.013236 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.013248 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.013250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.013402 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.013880 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.013884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.013886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.014195 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.014360 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.015221 (+ 861us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.015507 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260812 16:45:20.017069 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:19.992947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:38237)
0812 16:45:19.994114 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:19.994165 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:19.996376 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:19.997265 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:19.997276 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:19.998037 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:19.998825 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:19.998856 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.000071 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.000075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.002082 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.002117 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.003749 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.003760 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.004098 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.005070 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.005118 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.008107 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.012492 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.012503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.012521 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.013081 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.013522 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.013526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.013528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.013753 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.014346 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.014355 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.014782 (+ 427us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.015087 (+ 305us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.015717 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":595,"threads_started":1}
I20260812 16:45:20.019188 8775 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46243
I20260812 16:45:20.019538 8775 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:20.020390 8775 heartbeater.cc:511] Master 127.28.243.62:46243 requested a full tablet report, sending...
I20260812 16:45:20.022075 7971 ts_manager.cc:194] Registered new tserver with Master: 36ea8cbefec94008a079074cdee0247f (127.28.243.3:37727)
I20260812 16:45:20.023062 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:38237
I20260812 16:45:20.025599 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.243.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.61.keytab.
Entry for principal oryx/127.28.243.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.61.keytab.
I20260812 16:45:20.110414 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:45251
--webserver_interface=127.28.243.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.61.keytab
--principal=oryx/127.28.243.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46243,127.28.243.61:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:20.353322 8535 heartbeater.cc:503] Master 127.28.243.62:46243 was elected leader, sending a full tablet report...
W20260812 16:45:20.503264 8784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:20.503620 8784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:20.503760 8784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:20.511516 8784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:20.511682 8784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:20.511776 8784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:20.511857 8784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:20.511924 8784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:20.522689 8784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46243,127.28.243.61:45251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.61.keytab
--principal=oryx/127.28.243.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
W20260812 16:45:20.523190 8784 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.243.62:46243,127.28.243.61:45251'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 16:45:20.524436 8784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:20.525848 8784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:20.538251 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:20.538234 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:20.539007 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:20.539104 8784 server_base.cc:1034] running on GCE node
I20260812 16:45:20.540037 8784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:20.541746 8784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:20.543226 8784 hybrid_clock.cc:648] HybridClock initialized: now 1786553120543143 us; error 63 us; skew 500 ppm
I20260812 16:45:20.543628 8784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553120, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:20.547322 8784 init.cc:377] Logged in from keytab as oryx/127.28.243.61@KRBTEST.COM (short username oryx)
I20260812 16:45:20.549139 8784 webserver.cc:460] Webserver started at http://127.28.243.61:40983/ using document root <none> and password file <none>
I20260812 16:45:20.549971 8784 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:20.550088 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:20.550431 8784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:20.553797 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/instance:
uuid: "4ce105bc4ff9489a8b01d2d946a25964"
format_stamp: "Formatted at 2026-08-12 16:45:20 on dist-test-slave-1vmg"
server_key: "0149b6c3705486b7f90476da2bede37f"
server_key_iv: "95483349a2c0dcfa02180b0c28e97990"
server_key_version: "encryptionkey@0"
I20260812 16:45:20.554745 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal/instance:
uuid: "4ce105bc4ff9489a8b01d2d946a25964"
format_stamp: "Formatted at 2026-08-12 16:45:20 on dist-test-slave-1vmg"
server_key: "0149b6c3705486b7f90476da2bede37f"
server_key_iv: "95483349a2c0dcfa02180b0c28e97990"
server_key_version: "encryptionkey@0"
I20260812 16:45:20.561295 8784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:45:20.565482 8807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:20.567473 8784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:20.567695 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
uuid: "4ce105bc4ff9489a8b01d2d946a25964"
format_stamp: "Formatted at 2026-08-12 16:45:20 on dist-test-slave-1vmg"
server_key: "0149b6c3705486b7f90476da2bede37f"
server_key_iv: "95483349a2c0dcfa02180b0c28e97990"
server_key_version: "encryptionkey@0"
I20260812 16:45:20.567952 8784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:20.581423 8784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:20.586472 8784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:20.586789 8784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:20.609918 8784 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.61:45251
I20260812 16:45:20.609938 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.61:45251 every 8 connection(s)
I20260812 16:45:20.611549 8784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
I20260812 16:45:20.616526 8902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:20.616580 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 8784
I20260812 16:45:20.627485 8902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553120, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.61@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:20.646713 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.619112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.28.243.61:45251)
0812 16:45:20.620095 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.620108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.620159 (+ 51us) server_negotiation.cc:407] Connection header received
0812 16:45:20.621784 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.621809 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.622322 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.622733 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.624904 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.626749 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.630013 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.631323 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.631607 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.637817 (+ 6210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.637825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.637839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.637883 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.641878 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.642753 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.642761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.642770 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.643043 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.643529 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.643534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.643538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.644209 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.644524 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.645085 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.645593 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":443,"threads_started":1}
I20260812 16:45:20.652873 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.632208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:49874)
0812 16:45:20.633930 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:20.634015 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:20.634293 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:20.635037 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:20.635061 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:20.636123 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:20.636448 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.636460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.638092 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.638098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.639933 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.639957 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.640835 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.640847 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.641032 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.641783 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.641804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.644247 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.648773 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.648785 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.648793 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.649544 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.650448 (+ 904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.650453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.650455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.650635 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.651861 (+ 1226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.651869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.652064 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.652249 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.652560 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"thread_start_us":1117,"threads_started":1}
I20260812 16:45:20.653195 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.632277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.28.243.62:46243)
0812 16:45:20.633107 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.633114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.634176 (+ 1062us) server_negotiation.cc:407] Connection header received
0812 16:45:20.634462 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.634467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.634573 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.634694 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.636715 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.637913 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.640130 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.640600 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.640736 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.644490 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.644496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.644500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.644544 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.647673 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.649701 (+ 2028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.649707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.649710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.650132 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.650811 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.650816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.650819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.651641 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.651948 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.652569 (+ 621us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.652833 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":400,"threads_started":1}
I20260812 16:45:20.655117 8902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553120, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.61@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
I20260812 16:45:20.671778 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.656094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:48752)
0812 16:45:20.656303 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:20.656324 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:20.656448 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:20.656892 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:20.656895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:20.657285 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:20.657564 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.657578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.659225 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.659233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.660358 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.660380 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.661313 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.661319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.661505 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.662316 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.662341 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.664619 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.668364 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.668370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.668373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.669018 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.669668 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.669673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.669676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.669914 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.670693 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.670697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.670846 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.671119 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.671440 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 16:45:20.672093 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.656363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.28.243.61:45251)
0812 16:45:20.656589 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.656594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.656612 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:45:20.656657 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.656661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.656721 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.656820 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.657750 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.658846 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.660589 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.661054 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.661197 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.664857 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.664863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.664866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.664900 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.668107 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.669194 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.669199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.669202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.669394 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.670116 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.670121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.670123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.670507 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.670679 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.671221 (+ 542us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.671604 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":117}
I20260812 16:45:20.678745 8902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Bootstrap starting.
I20260812 16:45:20.682150 8902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:20.683353 8902 log.cc:824] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:20.686879 8902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: No bootstrap required, opened a new log
I20260812 16:45:20.697083 8902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } }
I20260812 16:45:20.697494 8902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:20.697582 8902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce105bc4ff9489a8b01d2d946a25964, State: Initialized, Role: FOLLOWER
I20260812 16:45:20.698283 8902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } }
I20260812 16:45:20.699612 8910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } } }
I20260812 16:45:20.700109 8910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:45:20.700834 8902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:45:20.711679 8931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:45:20.711903 8931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:45:20.717803 8784 master_runner.cc:428] Detected that this master 4ce105bc4ff9489a8b01d2d946a25964 is joining an existing cluster
I20260812 16:45:20.718014 8784 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.243.61:45251
I20260812 16:45:20.741216 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.724305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.28.243.62:46243)
0812 16:45:20.724495 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.724501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.726663 (+ 2162us) server_negotiation.cc:407] Connection header received
0812 16:45:20.726874 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.726880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.727045 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.727144 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.728693 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.729464 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.731223 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.731733 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.731874 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.734737 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.734744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.734747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.734790 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.737506 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.738493 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.738499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.738504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.738679 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.739238 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.739244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.739247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.739664 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.739822 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.740591 (+ 769us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.740862 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:45:20.741369 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.725138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:49876)
0812 16:45:20.726511 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:20.726529 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:20.726713 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:20.727212 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:20.727216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:20.728053 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:20.728508 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.728520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.729673 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.729678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.731039 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.731055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.732201 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.732213 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.732433 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.733367 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.733388 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.734477 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.737664 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.737670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.737674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.738314 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.738846 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.738852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.738856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.739096 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.739939 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.739944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.740277 (+ 333us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.740468 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.740833 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"mutex_wait_us":75,"spinlock_wait_cycles":34688,"thread_start_us":888,"threads_started":1}
I20260812 16:45:20.743283 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.724200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.28.243.61:45251)
0812 16:45:20.724496 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.724502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.725556 (+ 1054us) server_negotiation.cc:407] Connection header received
0812 16:45:20.725890 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.725897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.726018 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.726177 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.728381 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.729912 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.732417 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.733379 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.733519 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.736773 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.736780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.736785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.736823 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.739889 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.740716 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.740722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.740725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.740934 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.741388 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.741393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.741396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.741762 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.741936 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.742533 (+ 597us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.742877 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":93}
I20260812 16:45:20.743309 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.724307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:48756)
0812 16:45:20.725379 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:20.725403 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:20.725646 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:20.726424 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:20.726430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:20.727486 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:20.728150 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.728168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.730247 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.730253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.732198 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.732224 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.734191 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.734199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.734434 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.735339 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.735366 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.736518 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.740026 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.740032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.740037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.740591 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.741079 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.741084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.741088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.741251 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.742145 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.742148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.742280 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.742396 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.742723 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":32,"thread_start_us":642,"threads_started":1}
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.62@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
I20260812 16:45:20.773033 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.751943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:48766)
0812 16:45:20.752703 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:20.752771 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:20.752983 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:20.753805 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:20.753814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:20.754845 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:20.755245 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.755261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.756913 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.756919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.758350 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.758377 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.759661 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.759667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.759924 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.760728 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.760763 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.763786 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.768942 (+ 5156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.768966 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.768976 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.769657 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.770498 (+ 841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.770503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.770506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.770720 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.771646 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.771655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.771926 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.772194 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.772625 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":1001,"threads_started":1}
I20260812 16:45:20.773353 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.752132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.28.243.61:45251)
0812 16:45:20.752613 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.752620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.752866 (+ 246us) server_negotiation.cc:407] Connection header received
0812 16:45:20.753283 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.753288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.753418 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.753495 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.755486 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.756749 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.758559 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.759253 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.759379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.764065 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.764077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.764083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.764133 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.768700 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.769949 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.769959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.769963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.770268 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.770881 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.770888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.770892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.771477 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.771709 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.772437 (+ 728us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.772909 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260812 16:45:20.777199 7971 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.243.61:45251: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964"
member_type: NON_VOTER
last_known_addr {
host: "127.28.243.61"
port: 45251
}
attrs {
promote: true
}
}
dest_uuid: "20fa166f837c466a9f213e8d8c7d2cfd"
cas_config_opid_index: -1
I20260812 16:45:20.779569 7971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.28.243.60@KRBTEST.COM; defaulting to no policy
I20260812 16:45:20.784940 8957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4ce105bc4ff9489a8b01d2d946a25964 (127.28.243.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.60@KRBTEST.COM" created.
I20260812 16:45:20.787829 8864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:20.789968 8957 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.243.61:45251
I20260812 16:45:20.790810 8957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } }
I20260812 16:45:20.791069 8957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:20.792874 8864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Refusing update from remote peer 20fa166f837c466a9f213e8d8c7d2cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 16:45:20.796097 8784 master.cc:537] Master@127.28.243.61:45251 shutting down...
I20260812 16:45:20.799552 8956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:45:20.805457 7949 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 16:45:20.809166 7949 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd -> Peer 4ce105bc4ff9489a8b01d2d946a25964 (127.28.243.61:45251): Couldn't send request to peer 4ce105bc4ff9489a8b01d2d946a25964. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.60.keytab.
Entry for principal oryx/127.28.243.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.60.keytab.
I20260812 16:45:20.831365 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:33899
--webserver_interface=127.28.243.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.60.keytab
--principal=oryx/127.28.243.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:20.839771 8817 tablet.cc:2369] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:20.842608 8784 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:20.843274 8784 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:20.843533 8784 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: stopping tablet replica
I20260812 16:45:20.853574 8784 master.cc:552] Master@127.28.243.61:45251 shutdown complete.
I20260812 16:45:20.853798 8784 master_runner.cc:305] Clearing existing system tablet
I20260812 16:45:20.854347 8784 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:20.854611 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:20.858289 8784 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 16:45:20.860560 8972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:20.860997 8784 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:20.861876 8784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 16:45:20.862088 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
uuid: "4ce105bc4ff9489a8b01d2d946a25964"
format_stamp: "Formatted at 2026-08-12 16:45:20 on dist-test-slave-1vmg"
server_key: "0149b6c3705486b7f90476da2bede37f"
server_key_iv: "95483349a2c0dcfa02180b0c28e97990"
server_key_version: "encryptionkey@0"
I20260812 16:45:20.869621 8784 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:20.880013 8784 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 16:45:20.880421 8784 log.cc:1196] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 16:45:20.881006 8784 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Deleting consensus metadata
I20260812 16:45:20.881364 8784 master_runner.cc:315] Copying system tablet from 127.28.243.62:46243
I20260812 16:45:20.884791 8784 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: tablet copy: Beginning tablet copy session from remote peer at address 127.28.243.62:46243
I20260812 16:45:20.911582 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.886633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.28.243.62:46243)
0812 16:45:20.886883 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:20.886891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:20.891837 (+ 4946us) server_negotiation.cc:407] Connection header received
0812 16:45:20.892074 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:20.892081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:20.892179 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:20.892319 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:20.894868 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.896049 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.898139 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.899319 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.899578 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.903302 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:20.903311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:20.903316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:20.903372 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:20.907626 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.908700 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.908707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.908711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.908910 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:20.909447 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:20.909452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:20.909455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:20.909861 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:20.909999 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:20.910652 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 16:45:20.910977 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:45:20.912734 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:20.886230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:49890)
0812 16:45:20.891631 (+ 5401us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:20.891660 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:20.891895 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:20.892575 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:20.892582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:20.893526 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:20.894287 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.894322 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.896274 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.896281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.897929 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:20.897951 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:20.900031 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:20.900040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:20.900385 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:20.901511 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:20.901540 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:20.902942 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:20.907893 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.907904 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.907909 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.908510 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.909087 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:20.909092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:20.909096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:20.909293 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:20.910103 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:20.910109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:20.910326 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 16:45:20.911824 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0812 16:45:20.912208 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5195,"thread_start_us":5017,"threads_started":1}
I20260812 16:45:20.914228 7999 tablet_copy_service.cc:140] P 20fa166f837c466a9f213e8d8c7d2cfd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4ce105bc4ff9489a8b01d2d946a25964 ({username='oryx', principal='oryx/127.28.243.61@KRBTEST.COM'} at 127.0.0.1:49890)
I20260812 16:45:20.914793 7999 tablet_copy_service.cc:161] P 20fa166f837c466a9f213e8d8c7d2cfd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4ce105bc4ff9489a8b01d2d946a25964 at {username='oryx', principal='oryx/127.28.243.61@KRBTEST.COM'} at 127.0.0.1:49890: session id = 4ce105bc4ff9489a8b01d2d946a25964-00000000000000000000000000000000
I20260812 16:45:20.921955 7999 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 16:45:20.925719 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 16:45:20.938059 8784 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: tablet copy: Starting download of 0 data blocks...
I20260812 16:45:20.938743 8784 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: tablet copy: Starting download of 1 WAL segments...
I20260812 16:45:20.943928 8784 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 16:45:20.952272 7999 tablet_copy_service.cc:342] P 20fa166f837c466a9f213e8d8c7d2cfd: Request end of tablet copy session 4ce105bc4ff9489a8b01d2d946a25964-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.243.61@KRBTEST.COM'} at 127.0.0.1:49890
I20260812 16:45:20.952694 7999 tablet_copy_service.cc:433] P 20fa166f837c466a9f213e8d8c7d2cfd: ending tablet copy session 4ce105bc4ff9489a8b01d2d946a25964-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4ce105bc4ff9489a8b01d2d946a25964
I20260812 16:45:20.970000 8784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:20.973527 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:20.975687 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:20.982450 8984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:20.982615 8784 server_base.cc:1034] running on GCE node
I20260812 16:45:20.983253 8784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:20.984036 8784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:20.985275 8784 hybrid_clock.cc:648] HybridClock initialized: now 1786553120985184 us; error 68 us; skew 500 ppm
I20260812 16:45:20.985522 8784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:20 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553120, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:20.987870 8784 init.cc:377] Logged in from keytab as oryx/127.28.243.61@KRBTEST.COM (short username oryx)
I20260812 16:45:21.004328 8784 webserver.cc:460] Webserver started at http://127.28.243.61:38589/ using document root <none> and password file <none>
I20260812 16:45:21.004752 8784 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:21.004837 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:21.008749 8784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:45:21.011708 8995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:21.012328 8784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 16:45:21.012481 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
uuid: "4ce105bc4ff9489a8b01d2d946a25964"
format_stamp: "Formatted at 2026-08-12 16:45:20 on dist-test-slave-1vmg"
server_key: "0149b6c3705486b7f90476da2bede37f"
server_key_iv: "95483349a2c0dcfa02180b0c28e97990"
server_key_version: "encryptionkey@0"
I20260812 16:45:21.012668 8784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:21.026271 8784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:21.027383 8775 heartbeater.cc:503] Master 127.28.243.62:46243 was elected leader, sending a full tablet report...
I20260812 16:45:21.033481 8784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:21.063715 8784 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.61:45251
I20260812 16:45:21.063746 9081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.61:45251 every 8 connection(s)
I20260812 16:45:21.065045 8784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
I20260812 16:45:21.072307 9082 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:45:21.076871 9082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Bootstrap starting.
I20260812 16:45:21.101590 9082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:45:21.102310 9082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Bootstrap complete.
I20260812 16:45:21.103667 9082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } }
I20260812 16:45:21.103986 9082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ce105bc4ff9489a8b01d2d946a25964, State: Initialized, Role: LEARNER
I20260812 16:45:21.104287 9082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } }
I20260812 16:45:21.105572 9082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:21.105311 9086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } }
I20260812 16:45:21.105875 9086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: LEARNER
I20260812 16:45:21.119429 9107 catalog_manager.cc:1259] Loaded cluster ID: 56169dabf64d41baa83fea0e677ec7f5
I20260812 16:45:21.119561 9107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: loading cluster ID for follower catalog manager: success
I20260812 16:45:21.122437 9107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: acquiring CA information for follower catalog manager: success
I20260812 16:45:21.125218 9107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:45:21.280304 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.273575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:48768)
0812 16:45:21.273885 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.273906 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.274050 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.275061 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.275066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.275095 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:21.275299 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.275309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.276800 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.276805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.278103 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.278119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.279419 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.279426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.279665 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.279717 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.279831 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.279995 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260812 16:45:21.280402 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.273779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.28.243.61:45251)
0812 16:45:21.274663 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.274667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.274688 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:45:21.274755 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.274759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.274839 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.275018 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:21.275445 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.276626 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.278284 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.279200 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.279342 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.279613 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.279880 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.280035 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":599,"threads_started":1}
I20260812 16:45:21.281512 9043 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260812 16:45:21.282142 9086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20fa166f837c466a9f213e8d8c7d2cfd. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } }
I20260812 16:45:21.282454 9086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: LEARNER
W20260812 16:45:21.392787 8970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:21.393168 8970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:21.393486 8970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:21.403115 8970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:21.403260 8970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:21.403308 8970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:21.403431 8970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:21.403502 8970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:21.415112 8970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.60.keytab
--principal=oryx/127.28.243.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:21.416730 8970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:21.418864 8970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:21.435256 9119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:21.435362 9120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:21.435384 9125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:21.436537 8970 server_base.cc:1034] running on GCE node
I20260812 16:45:21.437686 8970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:21.439958 8970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:21.441270 8970 hybrid_clock.cc:648] HybridClock initialized: now 1786553121441147 us; error 61 us; skew 500 ppm
I20260812 16:45:21.441954 8970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553121, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:21.446774 8970 init.cc:377] Logged in from keytab as oryx/127.28.243.60@KRBTEST.COM (short username oryx)
I20260812 16:45:21.448989 8970 webserver.cc:460] Webserver started at http://127.28.243.60:46177/ using document root <none> and password file <none>
I20260812 16:45:21.449779 8970 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:21.449918 8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:21.450268 8970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:21.453646 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/instance:
uuid: "20c4175c0093422a815167264cbb2e5e"
format_stamp: "Formatted at 2026-08-12 16:45:21 on dist-test-slave-1vmg"
server_key: "b7b6b324aff245b434f22122d65ae29e"
server_key_iv: "6aebad0a6548a68ecf783f6fadae444d"
server_key_version: "encryptionkey@0"
I20260812 16:45:21.454537 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal/instance:
uuid: "20c4175c0093422a815167264cbb2e5e"
format_stamp: "Formatted at 2026-08-12 16:45:21 on dist-test-slave-1vmg"
server_key: "b7b6b324aff245b434f22122d65ae29e"
server_key_iv: "6aebad0a6548a68ecf783f6fadae444d"
server_key_version: "encryptionkey@0"
I20260812 16:45:21.461431 8970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260812 16:45:21.466590 9135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:21.468612 8970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:21.468827 8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
uuid: "20c4175c0093422a815167264cbb2e5e"
format_stamp: "Formatted at 2026-08-12 16:45:21 on dist-test-slave-1vmg"
server_key: "b7b6b324aff245b434f22122d65ae29e"
server_key_iv: "6aebad0a6548a68ecf783f6fadae444d"
server_key_version: "encryptionkey@0"
I20260812 16:45:21.469043 8970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:21.519485 8970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:21.525730 8970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:21.526042 8970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:21.549904 8970 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.60:33899
I20260812 16:45:21.549926 9230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.60:33899 every 8 connection(s)
I20260812 16:45:21.551581 8970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
I20260812 16:45:21.553839 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 8970
I20260812 16:45:21.558488 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:21.568092 9231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.60@KRBTEST.COM
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553121, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.60@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:21.586711 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.556977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.28.243.60:33899)
0812 16:45:21.558144 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.558158 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.558236 (+ 78us) server_negotiation.cc:407] Connection header received
0812 16:45:21.560227 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.560254 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.560693 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.561112 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.563184 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.564707 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.566722 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.567723 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.567878 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.574600 (+ 6722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.574614 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.574637 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.574709 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.580280 (+ 5571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.581745 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.581755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.581765 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.582031 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.582559 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.582565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.582568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.583420 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.583790 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.584368 (+ 578us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.584889 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":469,"threads_started":1}
I20260812 16:45:21.596721 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.572088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.28.243.62:46243)
0812 16:45:21.573983 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.573988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.574013 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:45:21.574456 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.574461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.574591 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.574712 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.577735 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.578791 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.581058 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.581903 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.582022 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.587425 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.587431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.587435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.587478 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.592446 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.593634 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.593639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.593642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.593834 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.594356 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.594360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.594362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.594735 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.594948 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.595699 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.596075 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1755,"thread_start_us":1615,"threads_started":1}
I20260812 16:45:21.596755 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.571903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:49902)
0812 16:45:21.573674 (+ 1771us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.573737 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.574117 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.575001 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.575021 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.576743 (+ 1722us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.577381 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.577406 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.579103 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.579110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.580833 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.580848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.582239 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.582246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.582434 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.583429 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.583451 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.587156 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.592705 (+ 5549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.592722 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.592730 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.593426 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.593976 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.593981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.593983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.594218 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.594906 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.594965 (+ 59us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.595288 (+ 323us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.595547 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.596076 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":620,"threads_started":1}
I20260812 16:45:21.598595 9231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553121, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.60@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
I20260812 16:45:21.615191 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.599535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:48784)
0812 16:45:21.599866 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.599883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.600041 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.600409 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.600413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.601048 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.601218 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.601228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.602339 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.602343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.603380 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.603393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.604122 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.604129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.604304 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.604911 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.604929 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.607889 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.611262 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.611269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.611273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.611841 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.612636 (+ 795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.612641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.612645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.613043 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.613841 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.613847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.614077 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.614245 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.614782 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 16:45:21.615195 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.599576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.28.243.61:45251)
0812 16:45:21.599781 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.599787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.599977 (+ 190us) server_negotiation.cc:407] Connection header received
0812 16:45:21.600175 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.600181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.600261 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.600366 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.601390 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.602184 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.603539 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.603963 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.604082 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.608198 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.608203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.608207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.608247 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.611088 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.612010 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.612015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.612017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.612375 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.613271 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.613277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.613279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.613615 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.613817 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.614480 (+ 663us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.614776 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:45:21.616546 9231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553121, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.60@KRBTEST.COM for oryx/127.28.243.60@KRBTEST.COM
I20260812 16:45:21.631860 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.617297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:49438)
0812 16:45:21.617541 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.617562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.617699 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.617981 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.617985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.618412 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.618568 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.618577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.619955 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.619962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.620956 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.620971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.622016 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.622023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.622226 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.622872 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.622891 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.625363 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.629341 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.629347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.629351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.629733 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.630095 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.630099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.630102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.630259 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.630792 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.630796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.630978 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.631120 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.631422 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 16:45:21.632040 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.617381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.28.243.60:33899)
0812 16:45:21.617548 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.617554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.617630 (+ 76us) server_negotiation.cc:407] Connection header received
0812 16:45:21.617804 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.617809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.617864 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.617952 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.618711 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.619710 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.621123 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.621762 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.621942 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.625581 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.625586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.625589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.625622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.629148 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.629849 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.629854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.629858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.629985 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.630362 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.630367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.630370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.630674 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.630814 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.631231 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.631664 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":65}
I20260812 16:45:21.638413 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Bootstrap starting.
I20260812 16:45:21.642424 9231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:21.643991 9231 log.cc:824] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:21.647117 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: No bootstrap required, opened a new log
I20260812 16:45:21.655897 9231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } }
I20260812 16:45:21.656311 9231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:21.656685 9231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c4175c0093422a815167264cbb2e5e, State: Initialized, Role: FOLLOWER
I20260812 16:45:21.657826 9231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } }
I20260812 16:45:21.659046 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } } }
I20260812 16:45:21.659584 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:45:21.660619 9231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:45:21.673666 9253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:45:21.673910 9253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:45:21.678097 8970 master_runner.cc:428] Detected that this master 20c4175c0093422a815167264cbb2e5e is joining an existing cluster
I20260812 16:45:21.678299 8970 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.243.60:33899
I20260812 16:45:21.695844 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 8079
I20260812 16:45:21.711398 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.683582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.28.243.61:45251)
0812 16:45:21.683790 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.683798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.686156 (+ 2358us) server_negotiation.cc:407] Connection header received
0812 16:45:21.686445 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.686450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.686539 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.686615 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.689135 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.690251 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.693699 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.694533 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.694664 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.703066 (+ 8402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.703074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.703080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.703131 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.707507 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.708499 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.708506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.708510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.708717 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.709234 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.709240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.709244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.709615 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.709778 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.710530 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.710949 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.1@KRBTEST.COM
I20260812 16:45:21.712759 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.684737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:48796)
0812 16:45:21.686035 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.686054 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.686273 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.686868 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.686872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.687775 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.688889 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.688907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.690723 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.690730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.693456 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.693473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.695401 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.695413 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.695729 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.697088 (+ 1359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.697121 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.702755 (+ 5634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.707752 (+ 4997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.707757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.707761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.708323 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.708886 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.708891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.708895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.709087 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.709868 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.709873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.710128 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.711767 (+ 1639us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.712181 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":200,"thread_start_us":3650,"threads_started":1}
I20260812 16:45:21.720417 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.683905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.28.243.62:46243)
0812 16:45:21.684203 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.684209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.685016 (+ 807us) server_negotiation.cc:407] Connection header received
0812 16:45:21.685358 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.685365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.685463 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.685549 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.689107 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.690502 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.693656 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.694345 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.694434 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.702444 (+ 8010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.702452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.702457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.702508 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.714484 (+ 11976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.717598 (+ 3114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.717605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.717608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.717835 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.718384 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.718389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.718392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.718861 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.719093 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.719751 (+ 658us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.720061 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:45:21.720494 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.685056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:49440)
0812 16:45:21.689061 (+ 4005us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.689083 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.689239 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.689679 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.689683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.690391 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.691627 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.691645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.693171 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.693176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.697664 (+ 4488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.697685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.699901 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.699911 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.700153 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.701062 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.701127 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.702400 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.716634 (+ 14234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.716642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.716648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.717645 (+ 997us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.718205 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.718210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.718214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.718374 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.719107 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.719111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.719309 (+ 198us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.719521 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.719916 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3544,"mutex_wait_us":26,"spinlock_wait_cycles":125952,"thread_start_us":819,"threads_started":1}
I20260812 16:45:21.721663 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.683638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:49904)
0812 16:45:21.684897 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.684917 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.685174 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.685837 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.685841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.686689 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.688827 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.688850 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.691255 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.691263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.693448 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.693460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.697362 (+ 3902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.697374 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.697689 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.699039 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.699069 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.702131 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.716642 (+ 14511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.716648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.716652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.717364 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.718028 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.718033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.718036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.718244 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.719183 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.719188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.719427 (+ 239us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.720746 (+ 1319us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.721150 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":247,"thread_start_us":741,"threads_started":1}
I20260812 16:45:21.721738 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.685635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.28.243.60:33899)
0812 16:45:21.685865 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.685871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.689149 (+ 3278us) server_negotiation.cc:407] Connection header received
0812 16:45:21.689357 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.689362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.689422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.689482 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.691816 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.692939 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.698033 (+ 5094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.699304 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.699459 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.703501 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.703507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.703512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.703553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.716371 (+ 12818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.717815 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.717820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.717824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.718056 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.718514 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.718518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.718521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.718971 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.720818 (+ 1847us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.720945 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.721342 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":70}
I20260812 16:45:21.725014 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.697230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.28.243.1:34481)
0812 16:45:21.699416 (+ 2186us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.699427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.699463 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:45:21.699546 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.699555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.699890 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.701954 (+ 2064us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.702295 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.703912 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.706174 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.707724 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.708671 (+ 947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.714256 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.714290 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.714323 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.714416 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.719298 (+ 4882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.720467 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.720478 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.720482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.720852 (+ 370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.721428 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.721433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.721437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.722567 (+ 1130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.724222 (+ 1655us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.724386 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.724682 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1968,"thread_start_us":1831,"threads_started":1}
I20260812 16:45:21.726420 8293 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.62@KRBTEST.COM for oryx/127.28.243.60@KRBTEST.COM
I20260812 16:45:21.749902 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.731546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:49452)
0812 16:45:21.732102 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.732127 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.732348 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.733130 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.733136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.733683 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.733912 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.734081 (+ 169us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.735932 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.735939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.737071 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.737088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.738282 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.738291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.738521 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.739261 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.739282 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.741620 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.745272 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.745279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.745283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.745846 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.746574 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.746585 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.746589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.746817 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.748232 (+ 1415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.748237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.748432 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.748747 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.749526 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373}
I20260812 16:45:21.751677 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.731471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.28.243.60:33899)
0812 16:45:21.731692 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.731699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.732530 (+ 831us) server_negotiation.cc:407] Connection header received
0812 16:45:21.732733 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.732740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.732937 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.734292 (+ 1355us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.734524 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.735745 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.737269 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.737803 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.738226 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.741903 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.741910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.741913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.741952 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.745074 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.746082 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.746090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.746093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.746349 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.747122 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.747128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.747131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.748060 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.748257 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.750768 (+ 2511us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.751330 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:45:21.754038 7971 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.243.60:33899: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20c4175c0093422a815167264cbb2e5e"
member_type: NON_VOTER
last_known_addr {
host: "127.28.243.60"
port: 33899
}
attrs {
promote: true
}
}
dest_uuid: "20fa166f837c466a9f213e8d8c7d2cfd"
cas_config_opid_index: 5
I20260812 16:45:21.755203 7971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } }
I20260812 16:45:21.757429 8956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 20c4175c0093422a815167264cbb2e5e (127.28.243.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.762521 9043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 LEARNER]: Refusing update from remote peer 20fa166f837c466a9f213e8d8c7d2cfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 16:45:21.765116 9185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:21.766657 9275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 16:45:21.769534 9185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Refusing update from remote peer 20fa166f837c466a9f213e8d8c7d2cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 16:45:21.771433 9275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:21.771761 8956 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.243.60:33899
I20260812 16:45:21.772521 8970 master.cc:537] Master@127.28.243.60:33899 shutting down...
W20260812 16:45:21.779058 9143 tablet.cc:2369] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:45:21.792797 7947 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd -> Peer 20c4175c0093422a815167264cbb2e5e (127.28.243.60:33899): Couldn't send request to peer 20c4175c0093422a815167264cbb2e5e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 16:45:21.795675 9043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 20c4175c0093422a815167264cbb2e5e (127.28.243.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.803414 9275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.803805 9275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:21.809891 9274 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: attempting to promote NON_VOTER 4ce105bc4ff9489a8b01d2d946a25964 to VOTER
I20260812 16:45:21.812286 9274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } }
I20260812 16:45:21.818943 8970 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:21.819631 8970 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:21.819981 8970 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: stopping tablet replica
I20260812 16:45:21.823714 9278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: NON_VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: true } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.826586 9278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: LEARNER
I20260812 16:45:21.830888 8970 master.cc:552] Master@127.28.243.60:33899 shutdown complete.
I20260812 16:45:21.831311 8970 master_runner.cc:305] Clearing existing system tablet
I20260812 16:45:21.831727 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.830648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:49458)
0812 16:45:21.831152 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.831376 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260812 16:45:21.832157 8970 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:21.832368 8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:21.833158 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.832631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:49464)
0812 16:45:21.832860 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.832996 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260812 16:45:21.833355 7947 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd -> Peer 20c4175c0093422a815167264cbb2e5e (127.28.243.60:33899): Couldn't send request to peer 20c4175c0093422a815167264cbb2e5e. Status: Network error: Client connection negotiation failed: client connection to 127.28.243.60:33899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 16:45:21.834434 9043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 LEARNER]: Refusing update from remote peer 20fa166f837c466a9f213e8d8c7d2cfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 16:45:21.835201 9275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 16:45:21.838681 8956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4ce105bc4ff9489a8b01d2d946a25964 (127.28.243.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.839510 9036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4ce105bc4ff9489a8b01d2d946a25964 (127.28.243.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.843899 9278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.844352 9278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:45:21.844457 9275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.844735 9275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:21.844669 8956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:21.845118 8956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:21.848115 8970 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.000s
I20260812 16:45:21.850298 9284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:21.850734 8970 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:21.851074 8970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 16:45:21.851189 8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
uuid: "20c4175c0093422a815167264cbb2e5e"
format_stamp: "Formatted at 2026-08-12 16:45:21 on dist-test-slave-1vmg"
server_key: "b7b6b324aff245b434f22122d65ae29e"
server_key_iv: "6aebad0a6548a68ecf783f6fadae444d"
server_key_version: "encryptionkey@0"
I20260812 16:45:21.856383 8970 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:21.863085 8970 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 16:45:21.863346 8970 log.cc:1196] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 16:45:21.863858 8970 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Deleting consensus metadata
I20260812 16:45:21.864130 8970 master_runner.cc:315] Copying system tablet from 127.28.243.62:46243
I20260812 16:45:21.867100 8970 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: tablet copy: Beginning tablet copy session from remote peer at address 127.28.243.62:46243
I20260812 16:45:21.877640 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 8079
I20260812 16:45:21.890606 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.868310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.28.243.62:46243)
0812 16:45:21.868696 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:21.868705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:21.869472 (+ 767us) server_negotiation.cc:407] Connection header received
0812 16:45:21.869767 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:21.869773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:21.869853 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:21.870081 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:21.871197 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.872308 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.874380 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.876234 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.876414 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.880835 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:21.880844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:21.880850 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:21.880908 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:21.885543 (+ 4635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.887245 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.887251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.887255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.887508 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:21.888200 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:21.888209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:21.888212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:21.888815 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:21.889059 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:21.889812 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 16:45:21.890163 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260812 16:45:21.890817 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:21.868306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:49916)
0812 16:45:21.869231 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:21.869250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:21.869562 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:21.870124 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:21.870129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:21.870608 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:21.870880 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.870891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.872492 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.872497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.874078 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:21.874110 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:21.876524 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:21.876534 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:21.876895 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:21.879165 (+ 2270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:21.879192 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:21.880496 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:21.885773 (+ 5277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.885778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.885781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.886840 (+ 1059us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.887655 (+ 815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:21.887661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:21.887665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:21.887845 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:21.889092 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:21.889100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:21.889432 (+ 332us) client_negotiation.cc:769] Sending connection context
0812 16:45:21.889623 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 16:45:21.890211 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":552,"threads_started":1}
I20260812 16:45:21.891834 7999 tablet_copy_service.cc:140] P 20fa166f837c466a9f213e8d8c7d2cfd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20c4175c0093422a815167264cbb2e5e ({username='oryx', principal='oryx/127.28.243.60@KRBTEST.COM'} at 127.0.0.1:49916)
I20260812 16:45:21.892030 7999 tablet_copy_service.cc:161] P 20fa166f837c466a9f213e8d8c7d2cfd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20c4175c0093422a815167264cbb2e5e at {username='oryx', principal='oryx/127.28.243.60@KRBTEST.COM'} at 127.0.0.1:49916: session id = 20c4175c0093422a815167264cbb2e5e-00000000000000000000000000000000
I20260812 16:45:21.894717 7999 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 16:45:21.897570 8970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 16:45:21.904670 8970 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: tablet copy: Starting download of 0 data blocks...
I20260812 16:45:21.905045 8970 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: tablet copy: Starting download of 1 WAL segments...
I20260812 16:45:21.914517 8970 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 16:45:21.916585 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:34481
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:46243
--webserver_port=35711
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:21.920112 7999 tablet_copy_service.cc:342] P 20fa166f837c466a9f213e8d8c7d2cfd: Request end of tablet copy session 20c4175c0093422a815167264cbb2e5e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.243.60@KRBTEST.COM'} at 127.0.0.1:49916
I20260812 16:45:21.920290 7999 tablet_copy_service.cc:433] P 20fa166f837c466a9f213e8d8c7d2cfd: ending tablet copy session 20c4175c0093422a815167264cbb2e5e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20c4175c0093422a815167264cbb2e5e
I20260812 16:45:21.935122 8970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:21.938131 9304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:21.939190 9307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:21.943980 9309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:21.944943 8970 server_base.cc:1034] running on GCE node
I20260812 16:45:21.945466 8970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:21.946730 8970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:21.948051 8970 hybrid_clock.cc:648] HybridClock initialized: now 1786553121947913 us; error 116 us; skew 500 ppm
I20260812 16:45:21.948419 8970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:21 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553121, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:21.951215 8970 init.cc:377] Logged in from keytab as oryx/127.28.243.60@KRBTEST.COM (short username oryx)
I20260812 16:45:21.959828 8970 webserver.cc:460] Webserver started at http://127.28.243.60:38573/ using document root <none> and password file <none>
I20260812 16:45:21.960275 8970 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:21.960407 8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:21.964314 8970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:45:21.967387 9322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:21.968060 8970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:45:21.968242 8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
uuid: "20c4175c0093422a815167264cbb2e5e"
format_stamp: "Formatted at 2026-08-12 16:45:21 on dist-test-slave-1vmg"
server_key: "b7b6b324aff245b434f22122d65ae29e"
server_key_iv: "6aebad0a6548a68ecf783f6fadae444d"
server_key_version: "encryptionkey@0"
I20260812 16:45:21.968456 8970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:21.993114 8970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:21.998163 8970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:22.039831 8970 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.60:33899
I20260812 16:45:22.039849 9417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.60:33899 every 8 connection(s)
I20260812 16:45:22.040850 8970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
I20260812 16:45:22.047739 9419 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:45:22.052502 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Bootstrap starting.
I20260812 16:45:22.076568 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:45:22.077239 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: Bootstrap complete.
I20260812 16:45:22.078545 9419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } }
I20260812 16:45:22.079007 9419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20c4175c0093422a815167264cbb2e5e, State: Initialized, Role: LEARNER
I20260812 16:45:22.079322 9419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } }
I20260812 16:45:22.086553 9419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:22.086162 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:22.087188 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: This master's current role is: LEARNER
I20260812 16:45:22.098069 9440 catalog_manager.cc:1259] Loaded cluster ID: 56169dabf64d41baa83fea0e677ec7f5
I20260812 16:45:22.098199 9440 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: loading cluster ID for follower catalog manager: success
I20260812 16:45:22.102043 9440 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: acquiring CA information for follower catalog manager: success
I20260812 16:45:22.105127 9440 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:45:22.231083 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.221572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.28.243.60:33899)
0812 16:45:22.222379 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:22.222385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:22.222411 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:45:22.222490 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:22.222495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:22.222587 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:22.222796 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:22.223304 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.225110 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.227684 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.229581 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.229765 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.230070 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:22.230269 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 16:45:22.230444 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":446,"threads_started":1}
I20260812 16:45:22.231925 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.221325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:49480)
0812 16:45:22.221620 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:22.221641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:22.221778 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:22.222849 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:22.222855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:22.222890 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:22.223114 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.223127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.225322 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.225328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.227299 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.227327 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.229844 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.229851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.230070 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.230110 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 16:45:22.230221 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 16:45:22.231066 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 16:45:22.233472 9374 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 16:45:22.234357 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20fa166f837c466a9f213e8d8c7d2cfd. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:22.234716 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: This master's current role is: LEARNER
W20260812 16:45:22.400028 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:22.400498 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:22.400665 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:22.409593 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:22.409749 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:22.409873 9303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:22.421497 9303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:34481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=35711
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:22.423166 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:22.424751 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:22.441006 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:22.441622 9455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:22.441919 9462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:22.442605 9303 server_base.cc:1034] running on GCE node
I20260812 16:45:22.443313 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:22.444384 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:22.445676 9303 hybrid_clock.cc:648] HybridClock initialized: now 1786553122445604 us; error 49 us; skew 500 ppm
I20260812 16:45:22.446054 9303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:22 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553122, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:22.451480 9303 init.cc:377] Logged in from keytab as oryx/127.28.243.1@KRBTEST.COM (short username oryx)
I20260812 16:45:22.453550 9303 webserver.cc:460] Webserver started at http://127.28.243.1:35711/ using document root <none> and password file <none>
I20260812 16:45:22.454485 9303 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:22.454720 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:22.461668 9303 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 16:45:22.465284 9470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:22.467300 9303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:22.467533 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "3a70ef06ada842c69ce906c08766ddcc"
format_stamp: "Formatted at 2026-08-12 16:45:18 on dist-test-slave-1vmg"
server_key: "90a142a8a566309628c2045c9d5c29f6"
server_key_iv: "6f4809421c1dae504220889bf16e3b2a"
server_key_version: "encryptionkey@0"
I20260812 16:45:22.468667 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:22.484879 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:22.491195 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:22.491744 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:22.493497 9303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:22.495885 9303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:22.496064 9303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:22.496269 9303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:22.496361 9303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:22.542630 9303 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:34481
I20260812 16:45:22.542711 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:34481 every 8 connection(s)
I20260812 16:45:22.544243 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:22.548849 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 9303
I20260812 16:45:22.549292 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 8314
Aug 12 16:45:22 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.2@KRBTEST.COM
Aug 12 16:45:22 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553122, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
Aug 12 16:45:22 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553122, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.60@KRBTEST.COM
I20260812 16:45:22.578504 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.551482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.28.243.2:39297)
0812 16:45:22.552233 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:22.552241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:22.552276 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:45:22.552362 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:22.552371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:22.552694 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:22.552913 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:22.556952 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.558818 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.560994 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.562262 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.562388 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.570593 (+ 8205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:22.570617 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:22.570640 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:22.570693 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:22.574271 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.575587 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.575597 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.575601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.575811 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.576387 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.576392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.576395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.576867 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:22.577186 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:22.577540 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 16:45:22.577865 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":732,"threads_started":1}
I20260812 16:45:22.580940 8527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:22.584241 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.547628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:39243 (local address 127.28.243.61:45251)
0812 16:45:22.550308 (+ 2680us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:22.550316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:22.552079 (+ 1763us) server_negotiation.cc:407] Connection header received
0812 16:45:22.556404 (+ 4325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:22.556412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:22.556583 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:22.556680 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:22.561537 (+ 4857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.562997 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.566826 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.567630 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.567759 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.573976 (+ 6217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:22.573987 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:22.573992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:22.574074 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:22.579570 (+ 5496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.580698 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.580705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.580709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.580901 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.581574 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.581580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.581584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.581963 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:22.582104 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:22.583272 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0812 16:45:22.583551 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2515,"thread_start_us":2376,"threads_started":1}
I20260812 16:45:22.587368 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.548404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.28.243.1:39243)
0812 16:45:22.551154 (+ 2750us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:22.551227 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:22.555887 (+ 4660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:22.557419 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:22.557434 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:22.558618 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:22.561176 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.561232 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.563322 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.563328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.566556 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.566580 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.568270 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.568279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.568600 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.569431 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:22.569458 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:22.573122 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:22.579787 (+ 6665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:22.579798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:22.579817 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:22.580508 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:22.581130 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:22.581135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:22.581139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:22.581428 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:22.582221 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:22.582231 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:22.582749 (+ 518us) client_negotiation.cc:769] Sending connection context
0812 16:45:22.584796 (+ 2047us) client_negotiation.cc:240] Negotiation successful
0812 16:45:22.585497 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2334,"thread_start_us":12992,"threads_started":1}
I20260812 16:45:22.588403 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.548402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:40239 (local address 127.28.243.60:33899)
0812 16:45:22.549006 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:22.549012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:22.561505 (+ 12493us) server_negotiation.cc:407] Connection header received
0812 16:45:22.562172 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:22.562178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:22.562270 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:22.562346 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:22.564065 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.565182 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.568841 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.569628 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.569800 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.577307 (+ 7507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:22.577315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:22.577321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:22.577370 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:22.581093 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.583088 (+ 1995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.583095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.583098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.583295 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.584349 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.584356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.584359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.584837 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:22.585044 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:22.587734 (+ 2690us) server_negotiation.cc:299] Negotiation successful
0812 16:45:22.588044 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460}
Aug 12 16:45:22 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553122, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:22.589473 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.548404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.28.243.1:40239)
0812 16:45:22.561384 (+ 12980us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:22.561409 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:22.561996 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:22.562630 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:22.562635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:22.563396 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:22.563869 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.563885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.565411 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.565416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.567626 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.567666 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.570395 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.570402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.570582 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.571459 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:22.571961 (+ 502us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:22.577011 (+ 5050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:22.582273 (+ 5262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:22.582277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:22.582280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:22.582894 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:22.584026 (+ 1132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:22.584031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:22.584034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:22.584189 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:22.587178 (+ 2989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:22.587183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:22.587390 (+ 207us) client_negotiation.cc:769] Sending connection context
0812 16:45:22.588667 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0812 16:45:22.589000 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12779,"mutex_wait_us":31,"spinlock_wait_cycles":49024,"thread_start_us":2010,"threads_started":1}
I20260812 16:45:22.589720 9683 heartbeater.cc:347] Connected to a master server at 127.28.243.61:45251
I20260812 16:45:22.590094 9683 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:22.591274 9683 heartbeater.cc:511] Master 127.28.243.61:45251 requested a full tablet report, sending...
I20260812 16:45:22.591845 9678 heartbeater.cc:347] Connected to a master server at 127.28.243.60:33899
I20260812 16:45:22.592296 9678 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:22.593214 9678 heartbeater.cc:511] Master 127.28.243.60:33899 requested a full tablet report, sending...
I20260812 16:45:22.595592 9018 ts_manager.cc:194] Registered new tserver with Master: 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481)
I20260812 16:45:22.597254 9355 ts_manager.cc:194] Registered new tserver with Master: 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481)
I20260812 16:45:22.600988 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.552035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:54845 (local address 127.28.243.62:46243)
0812 16:45:22.552512 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:22.552517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:22.571978 (+ 19461us) server_negotiation.cc:407] Connection header received
0812 16:45:22.572068 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:22.572074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:22.572176 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:22.572302 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:22.576714 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.577807 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.582263 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.583014 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.583160 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.591403 (+ 8243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:22.591415 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:22.591421 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:22.591497 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:22.596734 (+ 5237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.598006 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.598014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.598017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.598324 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:22.599015 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:22.599019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:22.599021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:22.599389 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:22.599541 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:22.600291 (+ 750us) server_negotiation.cc:299] Negotiation successful
0812 16:45:22.600626 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":778,"threads_started":1}
I20260812 16:45:22.601220 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:22.551829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:54845)
0812 16:45:22.571012 (+ 19183us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:22.571035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:22.571175 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:22.574436 (+ 3261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:22.574443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:22.575289 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:22.575789 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.575809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.580473 (+ 4664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.580479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.582078 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:22.582093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:22.583510 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:22.583517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:22.583696 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:22.585284 (+ 1588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:22.585306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:22.590874 (+ 5568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:22.597086 (+ 6212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:22.597094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:22.597098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:22.597763 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:22.598507 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:22.598515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:22.598520 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:22.598767 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:22.599534 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:22.599540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:22.599815 (+ 275us) client_negotiation.cc:769] Sending connection context
0812 16:45:22.600001 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 16:45:22.600565 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16730,"thread_start_us":16309,"threads_started":1}
I20260812 16:45:22.602375 9682 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46243
I20260812 16:45:22.602613 9682 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:22.603196 9682 heartbeater.cc:511] Master 127.28.243.62:46243 requested a full tablet report, sending...
I20260812 16:45:22.605332 7971 ts_manager.cc:194] Re-registered known tserver with Master: 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481)
I20260812 16:45:22.607178 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:54845
I20260812 16:45:22.679307 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 8314
I20260812 16:45:22.703959 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:39297
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:46243
--webserver_port=45641
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:22.739687 8956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: NON_VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: true } } }
I20260812 16:45:22.740084 8956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:22.741479 9700 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: attempting to promote NON_VOTER 20c4175c0093422a815167264cbb2e5e to VOTER
I20260812 16:45:22.742115 9700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } }
I20260812 16:45:22.744083 9374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 LEARNER]: Refusing update from remote peer 20fa166f837c466a9f213e8d8c7d2cfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 16:45:22.744165 9036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Refusing update from remote peer 20fa166f837c466a9f213e8d8c7d2cfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 16:45:22.744796 9700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 16:45:22.745253 8956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 16:45:22.748638 8956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 20c4175c0093422a815167264cbb2e5e (127.28.243.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.753003 8956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.753384 8956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:22.753679 8956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.754352 8956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:22.752348 9374 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 20c4175c0093422a815167264cbb2e5e (127.28.243.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.756115 9036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 20c4175c0093422a815167264cbb2e5e (127.28.243.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.762100 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.762557 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:22.763190 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.763581 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:45:22.764394 9711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:22.764825 9711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 16:45:23.105648 9699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:23.106007 9699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:23.106170 9699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:23.114439 9699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:23.114600 9699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:23.114748 9699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:23.127065 9699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:39297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=45641
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:23.128634 9699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:23.130189 9699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:23.145691 9723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:23.145771 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:23.145731 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:23.146505 9699 server_base.cc:1034] running on GCE node
I20260812 16:45:23.147296 9699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:23.148486 9699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:23.151949 9699 hybrid_clock.cc:648] HybridClock initialized: now 1786553123151837 us; error 90 us; skew 500 ppm
I20260812 16:45:23.152438 9699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:23.161381 9699 init.cc:377] Logged in from keytab as oryx/127.28.243.2@KRBTEST.COM (short username oryx)
I20260812 16:45:23.163654 9699 webserver.cc:460] Webserver started at http://127.28.243.2:45641/ using document root <none> and password file <none>
I20260812 16:45:23.164508 9699 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:23.164680 9699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:23.171427 9699 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:45:23.175275 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:23.177795 9699 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:45:23.178026 9699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "34b21184ad5e45e6bc9142e21ce3a705"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "8cf6e5bf144e4fbc8a444d619684e648"
server_key_iv: "fffd8817ea123c6af846522223ac0d5f"
server_key_version: "encryptionkey@0"
I20260812 16:45:23.179216 9699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:23.194306 9699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:23.200132 9699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:23.200470 9699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:23.201658 9699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:23.203310 9699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:23.203418 9699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:23.203529 9699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:23.203603 9699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:23.246611 9699 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:39297
I20260812 16:45:23.246673 9909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:39297 every 8 connection(s)
I20260812 16:45:23.248188 9699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:23.251972 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 9699
I20260812 16:45:23.252310 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 8547
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.3@KRBTEST.COM
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.60@KRBTEST.COM
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:23.297123 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.253472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.28.243.3:37727)
0812 16:45:23.266197 (+ 12725us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.266209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.266253 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:45:23.266363 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.266374 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.267104 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.267374 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.271178 (+ 3804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.273784 (+ 2606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.276024 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.276985 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.277149 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.284523 (+ 7374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.284546 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.284567 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.284634 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:23.291667 (+ 7033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.292703 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.292714 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.292718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.292921 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.293503 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.293508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.293512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.294029 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:23.295673 (+ 1644us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:23.295888 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 16:45:23.296206 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12133,"thread_start_us":11943,"threads_started":1}
I20260812 16:45:23.299918 8766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:23.301136 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.251300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:57573 (local address 127.28.243.61:45251)
0812 16:45:23.251790 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.251797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.271884 (+ 20087us) server_negotiation.cc:407] Connection header received
0812 16:45:23.275353 (+ 3469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.275359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.275460 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.275593 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.279486 (+ 3893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.280717 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.283846 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.284442 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.284550 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.291413 (+ 6863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.291421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.291426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.291488 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:23.295750 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.297116 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.297122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.297125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.297313 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.298687 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.298692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.298695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.299138 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:23.299286 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:23.300338 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0812 16:45:23.300721 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":1059,"threads_started":1}
I20260812 16:45:23.301353 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.271988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:49059 (local address 127.28.243.60:33899)
0812 16:45:23.272602 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.272608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.273583 (+ 975us) server_negotiation.cc:407] Connection header received
0812 16:45:23.275428 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.275435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.275574 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.275683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.279399 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.280550 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.283845 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.284504 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.284637 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.291373 (+ 6736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.291381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.291386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.291444 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:23.296044 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.297873 (+ 1829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.297880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.297884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.298051 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.298743 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.298750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.298754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.299334 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:23.299562 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:23.300404 (+ 842us) server_negotiation.cc:299] Negotiation successful
0812 16:45:23.300807 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":382,"threads_started":1}
I20260812 16:45:23.302690 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.251305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:58485 (local address 127.28.243.62:46243)
0812 16:45:23.268146 (+ 16841us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.268152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.268176 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:45:23.275428 (+ 7252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.275437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.275628 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.275750 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.279400 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.280550 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.283944 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.284681 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.284782 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.292751 (+ 7969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.292764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.292771 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.292962 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:23.296652 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.298224 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.298230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.298233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.298406 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.299364 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.299369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.299435 (+ 66us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.299826 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:23.299985 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:23.302058 (+ 2073us) server_negotiation.cc:299] Negotiation successful
0812 16:45:23.302333 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16692,"thread_start_us":16866,"threads_started":1}
I20260812 16:45:23.303102 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.251980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:58485)
0812 16:45:23.266596 (+ 14616us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:23.266699 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:23.274736 (+ 8037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:23.276350 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:23.276365 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:23.277827 (+ 1462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:23.279108 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.279141 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.280777 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.280783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.283625 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.283646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.285074 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.285079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.285613 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.286724 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:23.286776 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:23.291829 (+ 5053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:23.297530 (+ 5701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.297534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.297536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.298072 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.298681 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.298685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.298689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.298870 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.300480 (+ 1610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:23.300485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:23.300661 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 16:45:23.300806 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 16:45:23.301330 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14200,"mutex_wait_us":497,"spinlock_wait_cycles":55424,"thread_start_us":19091,"threads_started":1}
I20260812 16:45:23.303157 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.272620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.28.243.2:49059)
0812 16:45:23.273464 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:23.273481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:23.274736 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:23.276350 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:23.276365 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:23.277827 (+ 1462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:23.279108 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.279141 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.280777 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.280783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.283625 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.283646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.285062 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.285070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.285625 (+ 555us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.286614 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:23.286639 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:23.290974 (+ 4335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:23.296228 (+ 5254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.296234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.296237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.297468 (+ 1231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.298240 (+ 772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.298247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.298250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.298539 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.299678 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:23.299683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:23.299923 (+ 240us) client_negotiation.cc:769] Sending connection context
0812 16:45:23.301624 (+ 1701us) client_negotiation.cc:240] Negotiation successful
0812 16:45:23.302112 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":520,"spinlock_wait_cycles":43264,"thread_start_us":491,"threads_started":1}
I20260812 16:45:23.304102 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.251980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.28.243.2:57573)
0812 16:45:23.271659 (+ 19679us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:23.271680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:23.274735 (+ 3055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:23.276350 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:23.276365 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:23.277827 (+ 1462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:23.279245 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.279263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.280987 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.280993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.283624 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.283646 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.285059 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.285068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.285638 (+ 570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.286427 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:23.286475 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:23.290987 (+ 4512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:23.295969 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.295981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.296002 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.296959 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.298217 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.298221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.298223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.298550 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.299286 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:23.299296 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:23.299818 (+ 522us) client_negotiation.cc:769] Sending connection context
0812 16:45:23.300227 (+ 409us) client_negotiation.cc:240] Negotiation successful
0812 16:45:23.303697 (+ 3470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19498,"mutex_wait_us":540,"spinlock_wait_cycles":69760,"thread_start_us":13609,"threads_started":1}
I20260812 16:45:23.305063 9916 heartbeater.cc:347] Connected to a master server at 127.28.243.61:45251
I20260812 16:45:23.305148 9915 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46243
I20260812 16:45:23.305106 9912 heartbeater.cc:347] Connected to a master server at 127.28.243.60:33899
I20260812 16:45:23.305476 9915 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:23.305513 9912 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:23.305608 9916 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:23.307039 9915 heartbeater.cc:511] Master 127.28.243.62:46243 requested a full tablet report, sending...
I20260812 16:45:23.307081 9916 heartbeater.cc:511] Master 127.28.243.61:45251 requested a full tablet report, sending...
I20260812 16:45:23.307310 9912 heartbeater.cc:511] Master 127.28.243.60:33899 requested a full tablet report, sending...
I20260812 16:45:23.309340 7971 ts_manager.cc:194] Re-registered known tserver with Master: 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:23.309367 9018 ts_manager.cc:194] Registered new tserver with Master: 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:23.309351 9355 ts_manager.cc:194] Registered new tserver with Master: 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:23.310372 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:58485
I20260812 16:45:23.405143 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 8547
I20260812 16:45:23.432595 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:37727
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:46243
--webserver_port=33737
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:23.611315 9682 heartbeater.cc:503] Master 127.28.243.62:46243 was elected leader, sending a full tablet report...
W20260812 16:45:23.824352 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:23.824668 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:23.824792 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:23.833003 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:23.833178 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:23.833340 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:23.843773 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:37727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=33737
--tserver_master_addrs=127.28.243.62:46243,127.28.243.61:45251,127.28.243.60:33899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:23.845161 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:23.846519 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:23.861840 9946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:23.861910 9947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:23.862112 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:23.862879 9936 server_base.cc:1034] running on GCE node
I20260812 16:45:23.863720 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:23.864838 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:23.866111 9936 hybrid_clock.cc:648] HybridClock initialized: now 1786553123866035 us; error 74 us; skew 500 ppm
I20260812 16:45:23.866508 9936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:23.870193 9936 init.cc:377] Logged in from keytab as oryx/127.28.243.3@KRBTEST.COM (short username oryx)
I20260812 16:45:23.872350 9936 webserver.cc:460] Webserver started at http://127.28.243.3:33737/ using document root <none> and password file <none>
I20260812 16:45:23.873163 9936 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:23.873318 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:23.879951 9936 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:45:23.884330 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:23.886083 9936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 16:45:23.886272 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "36ea8cbefec94008a079074cdee0247f"
format_stamp: "Formatted at 2026-08-12 16:45:19 on dist-test-slave-1vmg"
server_key: "6575aade403d4895f532edde8184da6d"
server_key_iv: "440bae3a7a15aad41a383c5ae73ef597"
server_key_version: "encryptionkey@0"
I20260812 16:45:23.887485 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:23.904700 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:23.912381 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:23.913180 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:23.916222 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:23.917982 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:23.918092 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:23.918205 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:23.918283 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:23.957507 9936 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:37727
I20260812 16:45:23.957684 10134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:37727 every 8 connection(s)
I20260812 16:45:23.959226 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:23.966171 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 9936
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.60@KRBTEST.COM
Aug 12 16:45:23 dist-test-slave-1vmg krb5kdc[7897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553123, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.61@KRBTEST.COM
I20260812 16:45:24.000803 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.962806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:34235 (local address 127.28.243.62:46243)
0812 16:45:23.963546 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.963553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.965121 (+ 1568us) server_negotiation.cc:407] Connection header received
0812 16:45:23.969319 (+ 4198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.969668 (+ 349us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.969927 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.970055 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.976364 (+ 6309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.977637 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.981075 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.981750 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.981889 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.990126 (+ 8237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.990136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.990142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.990198 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:23.994474 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.995931 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.995939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.995944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.996176 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:23.997937 (+ 1761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:23.997943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:23.997946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:23.998469 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:23.998641 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.000122 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.000401 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":509,"threads_started":1}
I20260812 16:45:24.004920 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.963150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:41071 (local address 127.28.243.61:45251)
0812 16:45:23.963685 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.963689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.965122 (+ 1433us) server_negotiation.cc:407] Connection header received
0812 16:45:23.969149 (+ 4027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.969156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.969265 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.969890 (+ 625us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.981469 (+ 11579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.982548 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.987161 (+ 4613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.987860 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.988004 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.997547 (+ 9543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.997555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.997560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.997604 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.001706 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.002619 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.002625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.002629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.002828 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.003296 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.003300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.003303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.003681 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.003855 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.004250 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.004483 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
I20260812 16:45:24.004940 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.964033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.28.243.3:41071)
0812 16:45:23.964802 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:23.964891 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:23.968955 (+ 4064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:23.970737 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:23.970753 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:23.972980 (+ 2227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:23.981278 (+ 8298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.981293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.984344 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.984354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.987000 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.987019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.988058 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.988065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.988249 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.988825 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:23.988854 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:23.997306 (+ 8452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:24.001865 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:24.001870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:24.001875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:24.002450 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:24.002968 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:24.002973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:24.002976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:24.003159 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:24.003812 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:24.003817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:24.003984 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.004126 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.004459 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"spinlock_wait_cycles":60800,"thread_start_us":411,"threads_started":1}
I20260812 16:45:24.006971 10139 heartbeater.cc:347] Connected to a master server at 127.28.243.61:45251
I20260812 16:45:24.007647 10139 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:24.008740 10139 heartbeater.cc:511] Master 127.28.243.61:45251 requested a full tablet report, sending...
I20260812 16:45:24.003705 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.963585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:34235)
0812 16:45:23.964802 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:23.964898 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:23.968499 (+ 3601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:23.974389 (+ 5890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:23.974395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:23.975427 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:23.976032 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.976080 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.977950 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.977957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.980816 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.980842 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.982033 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.982046 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.982299 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.983160 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:23.983196 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:23.989729 (+ 6533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:23.994782 (+ 5053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.994802 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.994981 (+ 179us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.995750 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.997459 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:23.997464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:23.997467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:23.997798 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:23.998619 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:23.998630 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:23.999244 (+ 614us) client_negotiation.cc:769] Sending connection context
0812 16:45:23.999590 (+ 346us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.000249 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":463,"threads_started":1}
I20260812 16:45:24.011631 9018 ts_manager.cc:194] Registered new tserver with Master: 36ea8cbefec94008a079074cdee0247f (127.28.243.3:37727)
I20260812 16:45:24.012331 10137 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46243
I20260812 16:45:24.012511 10137 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:24.013253 10137 heartbeater.cc:511] Master 127.28.243.62:46243 requested a full tablet report, sending...
I20260812 16:45:24.014742 7971 ts_manager.cc:194] Re-registered known tserver with Master: 36ea8cbefec94008a079074cdee0247f (127.28.243.3:37727)
I20260812 16:45:24.015872 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.965610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:33993 (local address 127.28.243.60:33899)
0812 16:45:23.980832 (+ 15222us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.980837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.980856 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:45:23.980915 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.980919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.981000 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.981649 (+ 649us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.983043 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.984118 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.985912 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.986786 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.986968 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.998830 (+ 11862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:23.998838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:23.998843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:23.998891 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.004937 (+ 6046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.006858 (+ 1921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.006865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.006868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.007095 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.011824 (+ 4729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.011831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.011834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.013017 (+ 1183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.013210 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.015137 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.015447 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15089,"thread_start_us":15834,"threads_started":1}
I20260812 16:45:24.016036 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:34235
I20260812 16:45:24.016423 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.965419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.28.243.3:33993)
0812 16:45:23.978825 (+ 13406us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:23.978864 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:23.979302 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:23.981696 (+ 2394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:23.981700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:23.982415 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:23.982806 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.982821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.984296 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.984301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.985700 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:23.985719 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:23.987279 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:23.987286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:23.987465 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:23.988311 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:23.988355 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:23.997482 (+ 9127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:24.006116 (+ 8634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:24.006121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:24.006125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:24.006715 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:24.011331 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:24.011337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:24.011341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:24.011576 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:24.013303 (+ 1727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:24.013309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:24.013895 (+ 586us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.014077 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.014417 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13210,"thread_start_us":13558,"threads_started":1}
I20260812 16:45:24.017822 10135 heartbeater.cc:347] Connected to a master server at 127.28.243.60:33899
I20260812 16:45:24.018003 10135 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:24.018446 10135 heartbeater.cc:511] Master 127.28.243.60:33899 requested a full tablet report, sending...
I20260812 16:45:24.020629 9355 ts_manager.cc:194] Registered new tserver with Master: 36ea8cbefec94008a079074cdee0247f (127.28.243.3:37727)
I20260812 16:45:24.025952 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.985711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.28.243.61:45251)
0812 16:45:23.988667 (+ 2956us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.988673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.991102 (+ 2429us) server_negotiation.cc:407] Connection header received
0812 16:45:23.991729 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.991733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.991804 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.991922 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:23.995118 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.005726 (+ 10608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.010241 (+ 4515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.010907 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.011090 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.016959 (+ 5869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.016966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.016971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.017017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.022271 (+ 5254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.023261 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.023267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.023270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.023473 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.024043 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.024048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.024052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.024393 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.024627 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.025160 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.025427 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1733,"thread_start_us":1572,"threads_started":1}
I20260812 16:45:24.028235 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.986250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.28.243.62:46243)
0812 16:45:23.991474 (+ 5224us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:23.991480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:23.996954 (+ 5474us) server_negotiation.cc:407] Connection header received
0812 16:45:23.997153 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:23.997159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:23.997246 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:23.997357 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:24.005206 (+ 7849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.006751 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.009858 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.011067 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.011401 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.019680 (+ 8279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.019688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.019693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.019744 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.024014 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.024873 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.024879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.024882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.025057 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.026541 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.026546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.026550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.027064 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.027508 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.027744 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.027947 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5062,"thread_start_us":4925,"threads_started":1}
I20260812 16:45:24.033984 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:23.997819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.28.243.60:33899)
0812 16:45:24.009817 (+ 11998us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.009823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.009843 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:45:24.010199 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.010205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.010302 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.010421 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:24.018834 (+ 8413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.020030 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.022240 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.022791 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.022940 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.026365 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.026373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.026377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.026423 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.030106 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.031303 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.031308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.031312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.031519 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.031968 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.031973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.031977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.032359 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.033052 (+ 693us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.033233 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.033486 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":444,"threads_started":1}
I20260812 16:45:24.039846 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:24.047605 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:24.093937 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.078516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.28.243.1:34481)
0812 16:45:24.083606 (+ 5090us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.083617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.084404 (+ 787us) server_negotiation.cc:407] Connection header received
0812 16:45:24.084510 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.084516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.084734 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.084940 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:24.085936 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.087498 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.090978 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.092709 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.092901 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.093144 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.093450 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.093635 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4880,"thread_start_us":4771,"threads_started":1}
I20260812 16:45:24.094081 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.078219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:34481 (local address 127.0.0.1:57572)
0812 16:45:24.084095 (+ 5876us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:24.084116 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:24.084270 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:24.085219 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:24.085224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:24.085270 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:24.085700 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.085711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.087629 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.087633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.090704 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.090722 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.093046 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.093063 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.093265 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.093293 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.093378 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.093539 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5739,"mutex_wait_us":24,"thread_start_us":4271,"threads_started":1}
I20260812 16:45:24.095830 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.077132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:37727 (local address 127.0.0.1:35322)
0812 16:45:24.082815 (+ 5683us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:24.082840 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:24.083033 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:24.084734 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:24.084739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:24.084767 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:24.085225 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.085237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.087531 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.087536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.091294 (+ 3758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.091389 (+ 95us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.094859 (+ 3470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.094867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.095107 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.095145 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.095252 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.095406 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5529,"mutex_wait_us":40,"thread_start_us":6633,"threads_started":1}
I20260812 16:45:24.098824 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.083882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.28.243.2:39297)
0812 16:45:24.084514 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.084522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.086974 (+ 2452us) server_negotiation.cc:407] Connection header received
0812 16:45:24.087109 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.087118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.087390 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.088020 (+ 630us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:24.089342 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.091248 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.095531 (+ 4283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.097311 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.097451 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.097669 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.098403 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.098567 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":346,"threads_started":1}
I20260812 16:45:24.098872 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.084631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:39297 (local address 127.0.0.1:52334)
0812 16:45:24.086595 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:24.086614 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:24.086785 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:24.087794 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:24.087798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:24.087827 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:24.089041 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.089068 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.091724 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.091730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.095175 (+ 3445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.095189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.097915 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.097923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.098141 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.098177 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.098296 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.098458 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1833,"mutex_wait_us":56,"thread_start_us":1592,"threads_started":1}
I20260812 16:45:24.099493 9568 tablet_service.cc:1467] Processing CreateTablet for tablet bec2df092ed54c6a83a855d564c7e0c4 (DEFAULT_TABLE table=test-table [id=410bb876765d479787ae0b220ee5e277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:24.101298 9568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2df092ed54c6a83a855d564c7e0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:24.103307 9813 tablet_service.cc:1467] Processing CreateTablet for tablet bec2df092ed54c6a83a855d564c7e0c4 (DEFAULT_TABLE table=test-table [id=410bb876765d479787ae0b220ee5e277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:24.104568 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.077613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.28.243.3:37727)
0812 16:45:24.083862 (+ 6249us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.083870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.083948 (+ 78us) server_negotiation.cc:407] Connection header received
0812 16:45:24.084070 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.084081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.084436 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.084743 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:24.085400 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.087343 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.091767 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.094542 (+ 2775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.094722 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.094989 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.095471 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.103519 (+ 8048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6011,"thread_start_us":706,"threads_started":1}
I20260812 16:45:24.104945 9813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2df092ed54c6a83a855d564c7e0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:24.109833 10027 tablet_service.cc:1467] Processing CreateTablet for tablet bec2df092ed54c6a83a855d564c7e0c4 (DEFAULT_TABLE table=test-table [id=410bb876765d479787ae0b220ee5e277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:24.111831 10027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2df092ed54c6a83a855d564c7e0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:24.122902 10191 tablet_bootstrap.cc:492] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Bootstrap starting.
I20260812 16:45:24.125803 10193 tablet_bootstrap.cc:492] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Bootstrap starting.
I20260812 16:45:24.127911 10196 tablet_bootstrap.cc:492] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Bootstrap starting.
I20260812 16:45:24.128902 10191 tablet_bootstrap.cc:654] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:24.130115 10191 log.cc:824] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:24.130374 10193 tablet_bootstrap.cc:654] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:24.132871 10193 log.cc:824] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:24.134171 10196 tablet_bootstrap.cc:654] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:24.134395 10191 tablet_bootstrap.cc:492] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: No bootstrap required, opened a new log
I20260812 16:45:24.134791 10191 ts_tablet_manager.cc:1397] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260812 16:45:24.136214 10196 log.cc:824] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:24.137375 10193 tablet_bootstrap.cc:492] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: No bootstrap required, opened a new log
I20260812 16:45:24.137751 10193 ts_tablet_manager.cc:1397] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260812 16:45:24.141466 10196 tablet_bootstrap.cc:492] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: No bootstrap required, opened a new log
I20260812 16:45:24.142169 10196 ts_tablet_manager.cc:1397] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260812 16:45:24.144891 10191 raft_consensus.cc:348] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.145468 10191 raft_consensus.cc:374] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:24.145579 10191 raft_consensus.cc:729] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a70ef06ada842c69ce906c08766ddcc, State: Initialized, Role: FOLLOWER
I20260812 16:45:24.146423 10191 consensus_queue.cc:260] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.148767 10191 ts_tablet_manager.cc:1428] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260812 16:45:24.152415 10193 raft_consensus.cc:348] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.153030 10193 raft_consensus.cc:374] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:24.153160 10193 raft_consensus.cc:729] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b21184ad5e45e6bc9142e21ce3a705, State: Initialized, Role: FOLLOWER
I20260812 16:45:24.154040 10193 consensus_queue.cc:260] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.155159 10196 raft_consensus.cc:348] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.155550 9915 heartbeater.cc:503] Master 127.28.243.62:46243 was elected leader, sending a full tablet report...
I20260812 16:45:24.155635 10196 raft_consensus.cc:374] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:24.155740 10196 raft_consensus.cc:729] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ea8cbefec94008a079074cdee0247f, State: Initialized, Role: FOLLOWER
I20260812 16:45:24.156451 10193 ts_tablet_manager.cc:1428] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Time spent starting tablet: real 0.019s user 0.008s sys 0.011s
I20260812 16:45:24.156586 10196 consensus_queue.cc:260] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.159718 10196 ts_tablet_manager.cc:1428] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260812 16:45:24.160396 10137 heartbeater.cc:503] Master 127.28.243.62:46243 was elected leader, sending a full tablet report...
W20260812 16:45:24.213176 10142 tablet.cc:2369] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:45:24.252344 9918 tablet.cc:2369] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:24.255228 10208 raft_consensus.cc:482] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:24.255940 10208 raft_consensus.cc:504] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.258387 10208 leader_election.cc:288] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481), 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:24.267163 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.259059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:55917 (local address 127.28.243.1:34481)
0812 16:45:24.259264 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.259269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.259493 (+ 224us) server_negotiation.cc:407] Connection header received
0812 16:45:24.259701 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.259707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.259808 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.259903 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:24.260416 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.261317 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.263718 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.265575 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.265778 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.265860 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.266582 (+ 722us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.266701 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:45:24.267412 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.259682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:39297 (local address 127.28.243.3:52213)
0812 16:45:24.259911 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:24.259932 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:24.260023 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:24.260624 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:24.260630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:24.260655 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:24.260819 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.260829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.262017 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.262023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.264010 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.264022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.266247 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.266255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.266657 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.266701 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.266800 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.267183 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 16:45:24.267737 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.260185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:52213 (local address 127.28.243.2:39297)
0812 16:45:24.260359 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.260363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.260382 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:45:24.260444 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.260448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.260505 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.260591 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:24.260993 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.261876 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.264241 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.265970 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.266118 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.266172 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.266852 (+ 680us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.267181 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 16:45:24.267874 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.258959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:34481 (local address 127.28.243.3:55917)
0812 16:45:24.259391 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:24.259408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:24.259556 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:24.260068 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:24.260073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:24.260096 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:24.260279 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.260287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.261458 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.261463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.263509 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:24.263527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.265792 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.265798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:24.266214 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.266267 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 16:45:24.266573 (+ 306us) client_negotiation.cc:240] Negotiation successful
0812 16:45:24.266751 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260812 16:45:24.269201 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2df092ed54c6a83a855d564c7e0c4" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a70ef06ada842c69ce906c08766ddcc" is_pre_election: true
I20260812 16:45:24.269338 9840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2df092ed54c6a83a855d564c7e0c4" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b21184ad5e45e6bc9142e21ce3a705" is_pre_election: true
I20260812 16:45:24.269838 9599 raft_consensus.cc:2454] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 0.
I20260812 16:45:24.270164 9840 raft_consensus.cc:2454] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 0.
I20260812 16:45:24.270795 9963 leader_election.cc:302] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ea8cbefec94008a079074cdee0247f, 3a70ef06ada842c69ce906c08766ddcc; no voters:
I20260812 16:45:24.271394 10208 raft_consensus.cc:2781] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:24.271608 10208 raft_consensus.cc:482] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:24.271732 10208 raft_consensus.cc:3037] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:24.275627 10208 raft_consensus.cc:504] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.276662 10208 leader_election.cc:288] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 election: Requested vote from peers 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481), 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:24.277287 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2df092ed54c6a83a855d564c7e0c4" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a70ef06ada842c69ce906c08766ddcc"
I20260812 16:45:24.277531 9599 raft_consensus.cc:3037] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:24.277566 9840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2df092ed54c6a83a855d564c7e0c4" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b21184ad5e45e6bc9142e21ce3a705"
I20260812 16:45:24.277808 9840 raft_consensus.cc:3037] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:24.280336 9840 raft_consensus.cc:2454] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 1.
I20260812 16:45:24.280361 9599 raft_consensus.cc:2454] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 1.
I20260812 16:45:24.280884 9963 leader_election.cc:302] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b21184ad5e45e6bc9142e21ce3a705, 36ea8cbefec94008a079074cdee0247f; no voters:
I20260812 16:45:24.281271 10208 raft_consensus.cc:2781] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:24.282253 10208 raft_consensus.cc:686] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 1 LEADER]: Becoming Leader. State: Replica: 36ea8cbefec94008a079074cdee0247f, State: Running, Role: LEADER
I20260812 16:45:24.282953 10208 consensus_queue.cc:237] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } }
I20260812 16:45:24.289534 7970 catalog_manager.cc:5495] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ea8cbefec94008a079074cdee0247f (127.28.243.3). New cstate: current_term: 1 leader_uuid: "36ea8cbefec94008a079074cdee0247f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } health_report { overall_health: UNKNOWN } } }
W20260812 16:45:24.298249 9686 tablet.cc:2369] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:24.340420 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.333765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.28.243.3:37727)
0812 16:45:24.334017 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.334022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.334095 (+ 73us) server_negotiation.cc:407] Connection header received
0812 16:45:24.334288 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.334292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.334382 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.334544 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:24.334956 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.335821 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.337257 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.337689 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.337816 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.338885 (+ 1069us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:24.339569 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:24.339704 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.340082 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.340169 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 16:45:24.352299 9599 raft_consensus.cc:1261] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Refusing update from remote peer 36ea8cbefec94008a079074cdee0247f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:24.352294 9840 raft_consensus.cc:1261] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Refusing update from remote peer 36ea8cbefec94008a079074cdee0247f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:24.353674 10209 consensus_queue.cc:1035] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:24.354250 10208 consensus_queue.cc:1035] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [LEADER]: Connected to new peer: Peer: permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:24.403084 7970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:24.403517 7970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:24.411242 7970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20fa166f837c466a9f213e8d8c7d2cfd: Sending DeleteTablet for 3 replicas of tablet bec2df092ed54c6a83a855d564c7e0c4
I20260812 16:45:24.412725 10027 tablet_service.cc:1514] Processing DeleteTablet for tablet bec2df092ed54c6a83a855d564c7e0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:24 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:35322
I20260812 16:45:24.413085 9568 tablet_service.cc:1514] Processing DeleteTablet for tablet bec2df092ed54c6a83a855d564c7e0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:24 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:57572
I20260812 16:45:24.415591 9813 tablet_service.cc:1514] Processing DeleteTablet for tablet bec2df092ed54c6a83a855d564c7e0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:24 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:52334
I20260812 16:45:24.422784 10230 tablet_replica.cc:330] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: stopping tablet replica
I20260812 16:45:24.423610 10230 raft_consensus.cc:2229] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:24.424813 10230 raft_consensus.cc:2258] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:24.428397 10230 ts_tablet_manager.cc:1905] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:24.435824 10232 tablet_replica.cc:330] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: stopping tablet replica
I20260812 16:45:24.438369 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 7915
I20260812 16:45:24.439262 10232 raft_consensus.cc:2229] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:24.440169 10232 raft_consensus.cc:2258] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:24.442207 8018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:24.445376 10232 ts_tablet_manager.cc:1905] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:24.446100 10233 tablet_replica.cc:330] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: stopping tablet replica
I20260812 16:45:24.446519 10233 raft_consensus.cc:2229] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:24.447039 10233 raft_consensus.cc:2258] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:24.447386 10230 ts_tablet_manager.cc:1918] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:24.447643 10230 log.cc:1196] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/bec2df092ed54c6a83a855d564c7e0c4
I20260812 16:45:24.448374 10230 ts_tablet_manager.cc:1939] T bec2df092ed54c6a83a855d564c7e0c4 P 36ea8cbefec94008a079074cdee0247f: Deleting consensus metadata
I20260812 16:45:24.455900 10233 ts_tablet_manager.cc:1905] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:24.460043 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.418863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.28.243.61:45251)
0812 16:45:24.419092 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.419098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.435623 (+ 16525us) server_negotiation.cc:407] Connection header received
0812 16:45:24.436336 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.436343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.436441 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.436579 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:24.441452 (+ 4873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.442873 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.445536 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.446288 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.446409 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.450972 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.450980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.450986 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.451038 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.455074 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.457030 (+ 1956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.457037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.457041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.457261 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.457785 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.457791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.457795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.458220 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.458502 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.459339 (+ 837us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.459635 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:45:24.464663 10233 ts_tablet_manager.cc:1918] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:24.464897 10233 log.cc:1196] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/bec2df092ed54c6a83a855d564c7e0c4
I20260812 16:45:24.465590 10233 ts_tablet_manager.cc:1939] T bec2df092ed54c6a83a855d564c7e0c4 P 34b21184ad5e45e6bc9142e21ce3a705: Deleting consensus metadata
I20260812 16:45:24.473409 10232 ts_tablet_manager.cc:1918] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:24.473623 10232 log.cc:1196] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/bec2df092ed54c6a83a855d564c7e0c4
I20260812 16:45:24.474267 10232 ts_tablet_manager.cc:1939] T bec2df092ed54c6a83a855d564c7e0c4 P 3a70ef06ada842c69ce906c08766ddcc: Deleting consensus metadata
I20260812 16:45:24.484665 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.435978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.28.243.60:33899)
0812 16:45:24.436195 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.436201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.463235 (+ 27034us) server_negotiation.cc:407] Connection header received
0812 16:45:24.463552 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.463559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.463654 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.463730 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:24.466493 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.467684 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.469742 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.470402 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.470515 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.475430 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.475439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.475444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.475493 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.479373 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.480502 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.480510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.480514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.480930 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.482065 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.482073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.482077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.482814 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.483201 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.483708 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.483994 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:45:24.609241 7952 catalog_manager.cc:4841] TS 36ea8cbefec94008a079074cdee0247f (127.28.243.3:37727): tablet bec2df092ed54c6a83a855d564c7e0c4 (table test-table [id=410bb876765d479787ae0b220ee5e277]) successfully deleted
W20260812 16:45:24.609675 7952 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.243.3:37727, user_credentials={real_user=slave}} blocked reactor thread for 159095us
I20260812 16:45:24.610213 7949 catalog_manager.cc:4841] TS 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481): tablet bec2df092ed54c6a83a855d564c7e0c4 (table test-table [id=410bb876765d479787ae0b220ee5e277]) successfully deleted
I20260812 16:45:24.610493 7949 catalog_manager.cc:4841] TS 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297): tablet bec2df092ed54c6a83a855d564c7e0c4 (table test-table [id=410bb876765d479787ae0b220ee5e277]) successfully deleted
I20260812 16:45:24.611030 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 7915
I20260812 16:45:24.681906 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.659795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.28.243.61:45251)
0812 16:45:24.660002 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.660009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.661637 (+ 1628us) server_negotiation.cc:407] Connection header received
0812 16:45:24.661962 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.661970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.662070 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.662175 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:24.664990 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.666334 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.668935 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.669675 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.669845 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.674211 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.674220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.674225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.674299 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.678297 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.679467 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.679473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.679476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.679635 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.680114 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.680119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.680123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.680527 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.680805 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.681277 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.681525 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:45:24.685992 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:24.660993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.28.243.60:33899)
0812 16:45:24.661174 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:24.661179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:24.667907 (+ 6728us) server_negotiation.cc:407] Connection header received
0812 16:45:24.668865 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:24.668872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:24.669119 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:24.669420 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:24.671932 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.673051 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.674964 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:24.675726 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:24.675885 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:24.679192 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:24.679201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:24.679206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:24.679271 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:24.682626 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.683928 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.683933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.683935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.684071 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:24.684461 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:24.684464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:24.684466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:24.684832 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:24.685019 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:24.685480 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 16:45:24.685661 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:45:25.382472 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.381681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:56991)
0812 16:45:25.382141 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.382263 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
W20260812 16:45:25.383041 10137 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:46243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.387763 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.386651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:53571)
0812 16:45:25.387385 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.387537 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":548,"threads_started":1}
W20260812 16:45:25.388233 9682 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:46243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.389422 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.388666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:54971)
0812 16:45:25.389142 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.389257 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
W20260812 16:45:25.389837 9915 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:46243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.912724 10255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20fa166f837c466a9f213e8d8c7d2cfd)
I20260812 16:45:25.913240 10255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } }
I20260812 16:45:25.915062 10257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20fa166f837c466a9f213e8d8c7d2cfd)
I20260812 16:45:25.915468 10257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } }
I20260812 16:45:25.915712 10255 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243), 20c4175c0093422a815167264cbb2e5e (127.28.243.60:33899)
I20260812 16:45:25.917042 10257 leader_election.cc:288] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243), 4ce105bc4ff9489a8b01d2d946a25964 (127.28.243.61:45251)
I20260812 16:45:25.929512 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.917317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38086)
0812 16:45:25.928968 (+ 11651us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.929204 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11488,"thread_start_us":11722,"threads_started":1}
W20260812 16:45:25.929913 9323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.930387 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.916072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38076)
0812 16:45:25.929862 (+ 13790us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.930090 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13649,"thread_start_us":13511,"threads_started":1}
I20260812 16:45:25.933485 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.932158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38088)
0812 16:45:25.932330 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.932752 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 16:45:25.933573 8996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
W20260812 16:45:25.933916 9323 leader_election.cc:334] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243): Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.937785 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.937020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38094)
0812 16:45:25.937205 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.937384 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:45:25.938113 8996 leader_election.cc:334] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243): Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.942245 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.930088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:33899 (local address 127.0.0.1:44654)
0812 16:45:25.930603 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.930621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:25.930776 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:25.933655 (+ 2879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:25.933664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:25.933718 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:25.934392 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:25.934423 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.937660 (+ 3237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.937665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:25.939650 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:25.939668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.941223 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.941229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:25.941478 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:25.941524 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 16:45:25.941650 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 16:45:25.941881 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":1,"thread_start_us":2613,"threads_started":1}
I20260812 16:45:25.942464 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.930319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.28.243.60:33899)
0812 16:45:25.932498 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:25.932506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:25.932541 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:45:25.932752 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:25.932764 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:25.933281 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:25.934626 (+ 1345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:25.936059 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.937444 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.939913 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.941025 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.941170 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:25.941289 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:25.941880 (+ 591us) server_negotiation.cc:299] Negotiation successful
0812 16:45:25.942038 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":1824,"thread_start_us":1670,"threads_started":1}
I20260812 16:45:25.943156 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ce105bc4ff9489a8b01d2d946a25964" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20c4175c0093422a815167264cbb2e5e" is_pre_election: true
I20260812 16:45:25.943581 9374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce105bc4ff9489a8b01d2d946a25964 in term 1.
I20260812 16:45:25.944208 8996 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20c4175c0093422a815167264cbb2e5e, 4ce105bc4ff9489a8b01d2d946a25964; no voters: 20fa166f837c466a9f213e8d8c7d2cfd
I20260812 16:45:25.944440 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.917363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:45251 (local address 127.0.0.1:33028)
0812 16:45:25.929333 (+ 11970us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.929352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:25.929522 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:25.937099 (+ 7577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:25.937105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:25.937137 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:25.937496 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:25.937506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.939090 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.939097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:25.940858 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:25.940874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.943584 (+ 2710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.943590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:25.943752 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:25.943786 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:45:25.943881 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 16:45:25.944033 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11835,"mutex_wait_us":6,"thread_start_us":11262,"threads_started":1}
I20260812 16:45:25.944489 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.917998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.28.243.61:45251)
0812 16:45:25.936768 (+ 18770us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:25.936772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:25.936791 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:45:25.936855 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:25.936859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:25.936924 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:25.937034 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:25.937847 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.938885 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.941064 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:25.943225 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:25.943375 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:25.943485 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:25.943937 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 16:45:25.944080 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":837,"server-negotiator.queue_time_us":18628,"thread_start_us":18398,"threads_started":1}
I20260812 16:45:25.944638 10255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 16:45:25.944787 10255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 20fa166f837c466a9f213e8d8c7d2cfd)
I20260812 16:45:25.944918 10255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:45:25.945264 9036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20c4175c0093422a815167264cbb2e5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ce105bc4ff9489a8b01d2d946a25964" is_pre_election: true
I20260812 16:45:25.949182 10255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } }
I20260812 16:45:25.950493 9036 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20c4175c0093422a815167264cbb2e5e in current term 2: Already voted for candidate 4ce105bc4ff9489a8b01d2d946a25964 in this term.
I20260812 16:45:25.950605 10255 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [CANDIDATE]: Term 2 election: Requested vote from peers 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243), 20c4175c0093422a815167264cbb2e5e (127.28.243.60:33899)
I20260812 16:45:25.952649 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.951150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38110)
0812 16:45:25.951773 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.952297 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 16:45:25.952788 9326 leader_election.cc:302] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20c4175c0093422a815167264cbb2e5e; no voters: 20fa166f837c466a9f213e8d8c7d2cfd, 4ce105bc4ff9489a8b01d2d946a25964
I20260812 16:45:25.952886 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ce105bc4ff9489a8b01d2d946a25964" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20c4175c0093422a815167264cbb2e5e"
I20260812 16:45:25.953699 9374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:45:25.954965 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:25.954475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38116)
0812 16:45:25.954678 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:25.954768 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 16:45:25.955215 8996 leader_election.cc:334] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243): Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
I20260812 16:45:25.956970 9374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce105bc4ff9489a8b01d2d946a25964 in term 2.
I20260812 16:45:25.957196 10257 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:45:25.957679 8996 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20c4175c0093422a815167264cbb2e5e, 4ce105bc4ff9489a8b01d2d946a25964; no voters: 20fa166f837c466a9f213e8d8c7d2cfd
I20260812 16:45:25.958088 10255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:45:25.958640 10255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [term 2 LEADER]: Becoming Leader. State: Replica: 4ce105bc4ff9489a8b01d2d946a25964, State: Running, Role: LEADER
I20260812 16:45:25.959136 10255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } }
I20260812 16:45:25.961231 10277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce105bc4ff9489a8b01d2d946a25964. Latest consensus state: current_term: 2 leader_uuid: "4ce105bc4ff9489a8b01d2d946a25964" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:25.961634 10277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:25.962384 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:25.967605 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:25.968637 10279 catalog_manager.cc:1259] Loaded cluster ID: 56169dabf64d41baa83fea0e677ec7f5
I20260812 16:45:25.968732 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:25.970115 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:25.971096 10279 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Loaded TSK: 0
I20260812 16:45:25.972620 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:45:26.000504 9018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:26.009256 9018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:26.017503 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.016887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38124)
0812 16:45:26.017188 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.017308 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 16:45:26.017561 9374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [term 2 FOLLOWER]: Refusing update from remote peer 4ce105bc4ff9489a8b01d2d946a25964: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 16:45:26.018436 10255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 16:45:26.019130 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.018588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38134)
0812 16:45:26.018836 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.018972 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260812 16:45:26.019346 8996 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 -> Peer 20fa166f837c466a9f213e8d8c7d2cfd (127.28.243.62:46243): Couldn't send request to peer 20fa166f837c466a9f213e8d8c7d2cfd. Status: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 16:45:26.023010 10257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce105bc4ff9489a8b01d2d946a25964. Latest consensus state: current_term: 2 leader_uuid: "4ce105bc4ff9489a8b01d2d946a25964" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:26.023408 10257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:45:26.026189 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4ce105bc4ff9489a8b01d2d946a25964" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:26.028017 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:26.035970 10257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4ce105bc4ff9489a8b01d2d946a25964" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "20fa166f837c466a9f213e8d8c7d2cfd" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46243 } } peers { permanent_uuid: "4ce105bc4ff9489a8b01d2d946a25964" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 45251 } attrs { promote: false } } peers { permanent_uuid: "20c4175c0093422a815167264cbb2e5e" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 33899 } attrs { promote: false } } }
I20260812 16:45:26.036435 10257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c4175c0093422a815167264cbb2e5e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:45:26.066181 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.044220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:34481 (local address 127.0.0.1:60348)
0812 16:45:26.044496 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.044518 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:26.044635 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:26.059815 (+ 15180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:26.059821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:26.059858 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:26.060065 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:26.060078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.062059 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.062064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:26.063657 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:26.063671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.065379 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.065386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:26.065577 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.065619 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:45:26.065685 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 16:45:26.065839 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":3}
I20260812 16:45:26.067708 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.044700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:39297 (local address 127.0.0.1:53812)
0812 16:45:26.045073 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.045092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:26.045205 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:26.060648 (+ 15443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:26.060655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:26.060697 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:26.060995 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:26.061008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.062357 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.062362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:26.064629 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:26.064656 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.066405 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.066412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:26.066596 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.066635 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:45:26.066709 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:45:26.066880 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260812 16:45:26.067718 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.045755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.28.243.1:34481)
0812 16:45:26.059434 (+ 13679us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.059439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.059463 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:45:26.059540 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.059544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.059615 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.059744 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:26.060280 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.061867 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.063959 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.065146 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.066611 (+ 1465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.066730 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.066784 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.067162 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13545,"thread_start_us":13448,"threads_started":1}
I20260812 16:45:26.068699 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.045456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.28.243.2:39297)
0812 16:45:26.060291 (+ 14835us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.060297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.060317 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:45:26.060390 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.060396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.060472 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.060577 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:26.061178 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.062180 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.064911 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.066134 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.068003 (+ 1869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.068125 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.068192 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.068354 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14696,"thread_start_us":14604,"threads_started":1}
I20260812 16:45:26.069355 9568 tablet_service.cc:1467] Processing CreateTablet for tablet 117e1b26f0ee470e9a5055762a9f62a8 (DEFAULT_TABLE table=test-table [id=42f43e4b8a304b57a9af537ad0e0f487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:26.069957 9568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117e1b26f0ee470e9a5055762a9f62a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:26.070102 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.045250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:37727 (local address 127.0.0.1:37160)
0812 16:45:26.054137 (+ 8887us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.054161 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:26.054316 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:26.061634 (+ 7318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:26.061639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:26.061668 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:26.062169 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:26.062182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.063679 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.063685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:26.065989 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:26.066006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.068332 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.068339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:26.068550 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.068588 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:45:26.068724 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 16:45:26.069536 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8722,"mutex_wait_us":26,"thread_start_us":8371,"threads_started":1}
I20260812 16:45:26.071224 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.045746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.28.243.3:37727)
0812 16:45:26.061055 (+ 15309us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.061061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.061081 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:45:26.061152 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.061157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.061232 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.061344 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:26.062374 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.063487 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.066227 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.067842 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.067991 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.068154 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.070561 (+ 2407us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.070812 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15186,"thread_start_us":15057,"threads_started":1}
I20260812 16:45:26.072819 10027 tablet_service.cc:1467] Processing CreateTablet for tablet 117e1b26f0ee470e9a5055762a9f62a8 (DEFAULT_TABLE table=test-table [id=42f43e4b8a304b57a9af537ad0e0f487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:26.073655 10027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117e1b26f0ee470e9a5055762a9f62a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:26.075778 9813 tablet_service.cc:1467] Processing CreateTablet for tablet 117e1b26f0ee470e9a5055762a9f62a8 (DEFAULT_TABLE table=test-table [id=42f43e4b8a304b57a9af537ad0e0f487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:26.076327 9813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117e1b26f0ee470e9a5055762a9f62a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:26.089591 10294 tablet_bootstrap.cc:492] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Bootstrap starting.
I20260812 16:45:26.091130 10295 tablet_bootstrap.cc:492] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Bootstrap starting.
I20260812 16:45:26.093417 10295 tablet_bootstrap.cc:654] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:26.093417 10294 tablet_bootstrap.cc:654] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:26.095407 10296 tablet_bootstrap.cc:492] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Bootstrap starting.
I20260812 16:45:26.096491 10294 tablet_bootstrap.cc:492] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: No bootstrap required, opened a new log
I20260812 16:45:26.096701 10294 ts_tablet_manager.cc:1397] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260812 16:45:26.097658 10294 raft_consensus.cc:348] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.097941 10294 raft_consensus.cc:374] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:26.098044 10294 raft_consensus.cc:729] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ea8cbefec94008a079074cdee0247f, State: Initialized, Role: FOLLOWER
I20260812 16:45:26.098484 10295 tablet_bootstrap.cc:492] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: No bootstrap required, opened a new log
I20260812 16:45:26.098315 10294 consensus_queue.cc:260] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.098529 10296 tablet_bootstrap.cc:654] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:26.098658 10295 ts_tablet_manager.cc:1397] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 16:45:26.099318 10294 ts_tablet_manager.cc:1428] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 16:45:26.099573 10295 raft_consensus.cc:348] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.099850 10295 raft_consensus.cc:374] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:26.099972 10295 raft_consensus.cc:729] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a70ef06ada842c69ce906c08766ddcc, State: Initialized, Role: FOLLOWER
I20260812 16:45:26.100334 10295 consensus_queue.cc:260] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.104599 10295 ts_tablet_manager.cc:1428] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260812 16:45:26.106142 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.102418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:48481)
0812 16:45:26.105385 (+ 2967us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.105606 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2801,"thread_start_us":2593,"threads_started":1}
I20260812 16:45:26.108183 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.100110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:34169)
0812 16:45:26.107134 (+ 7024us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.107402 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6837,"threads_started":1}
I20260812 16:45:26.109196 9683 heartbeater.cc:503] Master 127.28.243.61:45251 was elected leader, sending a full tablet report...
I20260812 16:45:26.110805 10296 tablet_bootstrap.cc:492] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: No bootstrap required, opened a new log
I20260812 16:45:26.111078 10139 heartbeater.cc:503] Master 127.28.243.61:45251 was elected leader, sending a full tablet report...
I20260812 16:45:26.111126 10296 ts_tablet_manager.cc:1397] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260812 16:45:26.111833 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.111286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:34051)
0812 16:45:26.111524 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.111643 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260812 16:45:26.113780 10296 raft_consensus.cc:348] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.114110 10296 raft_consensus.cc:374] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:26.114230 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.108730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:58203)
0812 16:45:26.113762 (+ 5032us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.114029 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1864,"thread_start_us":4448,"threads_started":1}
I20260812 16:45:26.114218 10296 raft_consensus.cc:729] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b21184ad5e45e6bc9142e21ce3a705, State: Initialized, Role: FOLLOWER
I20260812 16:45:26.114508 10296 consensus_queue.cc:260] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.115658 10296 ts_tablet_manager.cc:1428] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260812 16:45:26.116734 10301 raft_consensus.cc:482] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:26.116981 10301 raft_consensus.cc:504] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.117242 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.116029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:47697)
0812 16:45:26.116692 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.116967 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":327,"threads_started":1}
I20260812 16:45:26.117820 10301 leader_election.cc:288] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481), 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:26.118256 9916 heartbeater.cc:503] Master 127.28.243.61:45251 was elected leader, sending a full tablet report...
I20260812 16:45:26.118386 9840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117e1b26f0ee470e9a5055762a9f62a8" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b21184ad5e45e6bc9142e21ce3a705" is_pre_election: true
I20260812 16:45:26.118377 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117e1b26f0ee470e9a5055762a9f62a8" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a70ef06ada842c69ce906c08766ddcc" is_pre_election: true
I20260812 16:45:26.118647 9840 raft_consensus.cc:2454] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 0.
I20260812 16:45:26.118639 9599 raft_consensus.cc:2454] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 0.
I20260812 16:45:26.118988 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.118501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:48569)
0812 16:45:26.118722 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.118808 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 16:45:26.119187 9963 leader_election.cc:302] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ea8cbefec94008a079074cdee0247f, 3a70ef06ada842c69ce906c08766ddcc; no voters:
I20260812 16:45:26.119488 10301 raft_consensus.cc:2781] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:26.119627 10301 raft_consensus.cc:482] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:26.119721 10301 raft_consensus.cc:3037] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:26.122303 10301 raft_consensus.cc:504] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.123044 10301 leader_election.cc:288] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 election: Requested vote from peers 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481), 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297)
I20260812 16:45:26.123871 9840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117e1b26f0ee470e9a5055762a9f62a8" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b21184ad5e45e6bc9142e21ce3a705"
I20260812 16:45:26.123754 9599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117e1b26f0ee470e9a5055762a9f62a8" candidate_uuid: "36ea8cbefec94008a079074cdee0247f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a70ef06ada842c69ce906c08766ddcc"
I20260812 16:45:26.124085 9840 raft_consensus.cc:3037] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:26.124316 9599 raft_consensus.cc:3037] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:26.126274 9840 raft_consensus.cc:2454] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 1.
I20260812 16:45:26.127112 9599 raft_consensus.cc:2454] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ea8cbefec94008a079074cdee0247f in term 1.
I20260812 16:45:26.127151 9963 leader_election.cc:302] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b21184ad5e45e6bc9142e21ce3a705, 36ea8cbefec94008a079074cdee0247f; no voters:
I20260812 16:45:26.128028 10301 raft_consensus.cc:2781] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:26.128666 10301 raft_consensus.cc:686] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 1 LEADER]: Becoming Leader. State: Replica: 36ea8cbefec94008a079074cdee0247f, State: Running, Role: LEADER
I20260812 16:45:26.129097 10301 consensus_queue.cc:237] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } }
I20260812 16:45:26.130208 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.129588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:36815)
0812 16:45:26.129868 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.129993 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 16:45:26.132513 9017 catalog_manager.cc:5495] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ea8cbefec94008a079074cdee0247f (127.28.243.3). New cstate: current_term: 1 leader_uuid: "36ea8cbefec94008a079074cdee0247f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ea8cbefec94008a079074cdee0247f" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260812 16:45:26.155329 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.147602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.28.243.3:37727)
0812 16:45:26.148010 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.148016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.148114 (+ 98us) server_negotiation.cc:407] Connection header received
0812 16:45:26.148416 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.148421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.148503 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.148595 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:26.149168 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.150288 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.152150 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.152687 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.152938 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.153649 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:26.153839 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:26.153941 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.154236 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.154366 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:45:26.159240 9599 raft_consensus.cc:1261] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Refusing update from remote peer 36ea8cbefec94008a079074cdee0247f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:26.159246 9840 raft_consensus.cc:1261] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Refusing update from remote peer 36ea8cbefec94008a079074cdee0247f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:26.160024 10322 consensus_queue.cc:1035] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [LEADER]: Connected to new peer: Peer: permanent_uuid: "34b21184ad5e45e6bc9142e21ce3a705" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:26.160411 10301 consensus_queue.cc:1035] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70ef06ada842c69ce906c08766ddcc" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 34481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:26.162865 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.162352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:40723)
0812 16:45:26.162540 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.162646 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 16:45:26.163414 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.162462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:33959)
0812 16:45:26.162960 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.163085 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
I20260812 16:45:26.177592 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.176125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.1:51749)
0812 16:45:26.177285 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.177406 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1022}
I20260812 16:45:26.178200 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.177437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.3:35211)
0812 16:45:26.177936 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.178037 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381}
I20260812 16:45:26.179188 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.178641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.28.243.2:38485)
0812 16:45:26.178878 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.179003 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260812 16:45:26.186136 9017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:26.186447 9017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:26.195279 9017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ce105bc4ff9489a8b01d2d946a25964: Sending DeleteTablet for 3 replicas of tablet 117e1b26f0ee470e9a5055762a9f62a8
I20260812 16:45:26.197090 9568 tablet_service.cc:1514] Processing DeleteTablet for tablet 117e1b26f0ee470e9a5055762a9f62a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:26 UTC) from {username='oryx', principal='oryx/127.28.243.61@KRBTEST.COM'} at 127.0.0.1:60348
I20260812 16:45:26.197127 9813 tablet_service.cc:1514] Processing DeleteTablet for tablet 117e1b26f0ee470e9a5055762a9f62a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:26 UTC) from {username='oryx', principal='oryx/127.28.243.61@KRBTEST.COM'} at 127.0.0.1:53812
I20260812 16:45:26.197758 10027 tablet_service.cc:1514] Processing DeleteTablet for tablet 117e1b26f0ee470e9a5055762a9f62a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:26 UTC) from {username='oryx', principal='oryx/127.28.243.61@KRBTEST.COM'} at 127.0.0.1:37160
I20260812 16:45:26.197916 10349 tablet_replica.cc:330] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: stopping tablet replica
I20260812 16:45:26.198127 10349 raft_consensus.cc:2229] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:26.198488 10349 raft_consensus.cc:2258] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:26.198652 10351 tablet_replica.cc:330] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: stopping tablet replica
I20260812 16:45:26.199158 10351 raft_consensus.cc:2229] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:26.199679 10351 raft_consensus.cc:2258] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:26.199824 10349 ts_tablet_manager.cc:1905] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:26.201139 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 9303
I20260812 16:45:26.201309 10351 ts_tablet_manager.cc:1905] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:26.205729 10350 tablet_replica.cc:330] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: stopping tablet replica
I20260812 16:45:26.206115 10350 raft_consensus.cc:2229] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:26.206848 10350 raft_consensus.cc:2258] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:26.210103 10351 ts_tablet_manager.cc:1918] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:26.210099 10349 ts_tablet_manager.cc:1918] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:26.210502 10351 log.cc:1196] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/117e1b26f0ee470e9a5055762a9f62a8
I20260812 16:45:26.210500 10349 log.cc:1196] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/117e1b26f0ee470e9a5055762a9f62a8
I20260812 16:45:26.211318 10351 ts_tablet_manager.cc:1939] T 117e1b26f0ee470e9a5055762a9f62a8 P 36ea8cbefec94008a079074cdee0247f: Deleting consensus metadata
I20260812 16:45:26.211318 10349 ts_tablet_manager.cc:1939] T 117e1b26f0ee470e9a5055762a9f62a8 P 34b21184ad5e45e6bc9142e21ce3a705: Deleting consensus metadata
I20260812 16:45:26.213701 10350 ts_tablet_manager.cc:1905] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:26.213639 9001 catalog_manager.cc:4841] TS 36ea8cbefec94008a079074cdee0247f (127.28.243.3:37727): tablet 117e1b26f0ee470e9a5055762a9f62a8 (table test-table [id=42f43e4b8a304b57a9af537ad0e0f487]) successfully deleted
I20260812 16:45:26.220393 10350 ts_tablet_manager.cc:1918] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:26.220521 10350 log.cc:1196] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/117e1b26f0ee470e9a5055762a9f62a8
I20260812 16:45:26.220988 10350 ts_tablet_manager.cc:1939] T 117e1b26f0ee470e9a5055762a9f62a8 P 3a70ef06ada842c69ce906c08766ddcc: Deleting consensus metadata
I20260812 16:45:26.221920 8997 catalog_manager.cc:4841] TS 34b21184ad5e45e6bc9142e21ce3a705 (127.28.243.2:39297): tablet 117e1b26f0ee470e9a5055762a9f62a8 (table test-table [id=42f43e4b8a304b57a9af537ad0e0f487]) successfully deleted
I20260812 16:45:26.222232 8997 catalog_manager.cc:4841] TS 3a70ef06ada842c69ce906c08766ddcc (127.28.243.1:34481): tablet 117e1b26f0ee470e9a5055762a9f62a8 (table test-table [id=42f43e4b8a304b57a9af537ad0e0f487]) successfully deleted
I20260812 16:45:26.222525 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.202386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.28.243.1:34481)
0812 16:45:26.202626 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.202632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.204395 (+ 1763us) server_negotiation.cc:407] Connection header received
0812 16:45:26.204584 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.204590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.204663 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.204753 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:26.206694 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.208250 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.209944 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.210506 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.210668 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.215486 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:26.215517 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:26.215550 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:26.215643 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:26.219285 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:26.220162 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:26.220171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:26.220174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:26.220327 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:26.220739 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:26.220744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:26.220767 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:26.221214 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:26.221520 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.221952 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.222223 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:45:26.223300 9665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:26.343380 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 9303
I20260812 16:45:26.378973 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 9699
I20260812 16:45:26.395612 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.379806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.28.243.2:39297)
0812 16:45:26.380031 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.380037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.380058 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:45:26.380172 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.380178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.380266 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.380353 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:26.381712 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.382423 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.383883 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.384353 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.384479 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.388195 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:26.388213 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:26.388234 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:26.388297 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:26.391409 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:26.392460 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:26.392469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:26.392471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:26.392642 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:26.393071 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:26.393077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:26.393080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:26.393492 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:26.393782 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.394606 (+ 824us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.395188 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:45:26.396546 9903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:26.498581 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 9699
I20260812 16:45:26.539893 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 9936
I20260812 16:45:26.555003 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.540676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.28.243.3:37727)
0812 16:45:26.540841 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:26.540845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:26.540865 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:45:26.540934 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:26.540945 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:26.541025 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:26.541134 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:26.542530 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.543428 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.544640 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:26.545066 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:26.545168 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:26.547983 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:26.547999 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:26.548018 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:26.548074 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:26.551397 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:26.552241 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:26.552250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:26.552253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:26.552414 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:26.552931 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:26.553001 (+ 70us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:26.553004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:26.553454 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:26.553719 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:26.554095 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 16:45:26.554516 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 16:45:26.555913 10127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:26.637347 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.636457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38148)
0812 16:45:26.637035 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.637158 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":334,"threads_started":1}
I20260812 16:45:26.638898 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:26.638480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46243 (local address 127.0.0.1:38156)
0812 16:45:26.638670 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:26.638769 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:45:26.698361 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 9936
I20260812 16:45:26.734566 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 8784
I20260812 16:45:26.736188 9073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:26.877784 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 8784
I20260812 16:45:26.906888 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 8970
I20260812 16:45:26.908218 9410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:27.033881 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 8970
2026-08-12T16:45:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11430 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:45:27 dist-test-slave-1vmg krb5kdc[10365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:45:27 dist-test-slave-1vmg krb5kdc[10365](info): set up 2 sockets
Aug 12 16:45:27 dist-test-slave-1vmg krb5kdc[10365](info): commencing operation
krb5kdc: starting...
W20260812 16:45:28.395910 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:45:28 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab.
Entry for principal oryx/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab.
I20260812 16:45:28.724759 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:44101
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab
--principal=oryx/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:44101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:29.108798 10385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:29.109169 10385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:29.109308 10385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:29.116955 10385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:29.117110 10385 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 16:45:29.117192 10385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:29.117281 10385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:29.117354 10385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:29.117439 10385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:29.129731 10385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:44101
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.62.keytab
--principal=oryx/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:44101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:29.131253 10385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:29.132730 10385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:29.147630 10391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:29.147564 10392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:29.147588 10396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:29.149201 10385 server_base.cc:1034] running on GCE node
I20260812 16:45:29.150256 10385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:29.152164 10385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:29.155587 10385 hybrid_clock.cc:648] HybridClock initialized: now 1786553129155457 us; error 78 us; skew 500 ppm
I20260812 16:45:29.156085 10385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:29 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:29.165457 10385 init.cc:377] Logged in from keytab as oryx/127.28.243.62@KRBTEST.COM (short username oryx)
I20260812 16:45:29.168192 10385 webserver.cc:460] Webserver started at http://127.28.243.62:40105/ using document root <none> and password file <none>
I20260812 16:45:29.169067 10385 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:29.169190 10385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:29.169512 10385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:29.172956 10385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "66c8c75fb558491a8313ab69e884574c"
format_stamp: "Formatted at 2026-08-12 16:45:29 on dist-test-slave-1vmg"
server_key: "35707bdea8baa1992c0ef475a78e1db5"
server_key_iv: "351a64988f294968659f73dc0e1e5f9d"
server_key_version: "encryptionkey@0"
I20260812 16:45:29.173843 10385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "66c8c75fb558491a8313ab69e884574c"
format_stamp: "Formatted at 2026-08-12 16:45:29 on dist-test-slave-1vmg"
server_key: "35707bdea8baa1992c0ef475a78e1db5"
server_key_iv: "351a64988f294968659f73dc0e1e5f9d"
server_key_version: "encryptionkey@0"
I20260812 16:45:29.180588 10385 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:45:29.184906 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:29.186583 10385 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 16:45:29.186836 10385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "66c8c75fb558491a8313ab69e884574c"
format_stamp: "Formatted at 2026-08-12 16:45:29 on dist-test-slave-1vmg"
server_key: "35707bdea8baa1992c0ef475a78e1db5"
server_key_iv: "351a64988f294968659f73dc0e1e5f9d"
server_key_version: "encryptionkey@0"
I20260812 16:45:29.187306 10385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:29.219290 10385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:29.225080 10385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:29.225358 10385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:29.251006 10385 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:44101
I20260812 16:45:29.251024 10507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:44101 every 8 connection(s)
I20260812 16:45:29.252557 10385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:29.257656 10510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:29.259919 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 10385
I20260812 16:45:29.260272 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:29.261144 29644 external_mini_cluster.cc:1428] Setting key 1f5a51f482908bb30624de5f8da4379f
I20260812 16:45:29.271317 10510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Bootstrap starting.
I20260812 16:45:29.274705 10510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Neither blocks nor log segments found. Creating new log.
Aug 12 16:45:29 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:29.275796 10510 log.cc:824] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:29.279501 10510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: No bootstrap required, opened a new log
I20260812 16:45:29.286882 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:29.263719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.28.243.62:44101)
0812 16:45:29.264605 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:29.264622 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:29.264670 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:45:29.266510 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:29.266538 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:29.267056 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:29.267477 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:29.269053 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:29.270449 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:29.271879 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:29.272583 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:29.272787 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:29.277022 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:29.277031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:29.277047 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:29.277090 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:29.281357 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:29.282272 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:29.282283 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:29.282293 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:29.282546 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:29.283083 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:29.283088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:29.283091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:29.283935 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:29.284257 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:29.284789 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 16:45:29.285331 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":394,"threads_started":1}
I20260812 16:45:29.290849 10510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c8c75fb558491a8313ab69e884574c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44101 } }
I20260812 16:45:29.291446 10510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:29.291534 10510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c8c75fb558491a8313ab69e884574c, State: Initialized, Role: FOLLOWER
I20260812 16:45:29.292297 10510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c8c75fb558491a8313ab69e884574c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44101 } }
I20260812 16:45:29.292702 10510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:29.292856 10510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:29.293138 10510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:29.295864 10510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c8c75fb558491a8313ab69e884574c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44101 } }
I20260812 16:45:29.296473 10510 leader_election.cc:302] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c8c75fb558491a8313ab69e884574c; no voters:
I20260812 16:45:29.297158 10510 leader_election.cc:288] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:29.297266 10519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:29.299201 10519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [term 1 LEADER]: Becoming Leader. State: Replica: 66c8c75fb558491a8313ab69e884574c, State: Running, Role: LEADER
I20260812 16:45:29.299389 10510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:29.299777 10519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c8c75fb558491a8313ab69e884574c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44101 } }
I20260812 16:45:29.303009 10526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c8c75fb558491a8313ab69e884574c. Latest consensus state: current_term: 1 leader_uuid: "66c8c75fb558491a8313ab69e884574c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c8c75fb558491a8313ab69e884574c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44101 } } }
I20260812 16:45:29.303109 10520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c8c75fb558491a8313ab69e884574c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c8c75fb558491a8313ab69e884574c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44101 } } }
I20260812 16:45:29.303483 10526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:29.304920 10520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:29.305261 10545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:29.315099 10545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:29.327823 10545 catalog_manager.cc:1347] Generated new cluster ID: 7a7dac270eab4635a63d3d25c16b39b0
I20260812 16:45:29.327982 10545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:29.354951 10545 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:29.356993 10545 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:29.378439 10545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Generated new TSK 0
I20260812 16:45:29.379806 10545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab.
Entry for principal oryx/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab.
I20260812 16:45:29.492616 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44101
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:29.901342 10566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:29.901657 10566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:29.901767 10566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:29.909723 10566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:29.909894 10566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:29.909987 10566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:45:29.910070 10566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:29.920655 10566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:44101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:29.922446 10566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:29.923949 10566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:29.941232 10574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:29.941393 10575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:29.943650 10577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:29.943996 10566 server_base.cc:1034] running on GCE node
I20260812 16:45:29.945053 10566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:29.946621 10566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:29.947988 10566 hybrid_clock.cc:648] HybridClock initialized: now 1786553129947858 us; error 85 us; skew 500 ppm
I20260812 16:45:29.948414 10566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:29 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:29.952307 10566 init.cc:377] Logged in from keytab as oryx/127.28.243.1@KRBTEST.COM (short username oryx)
I20260812 16:45:29.954285 10566 webserver.cc:460] Webserver started at http://127.28.243.1:39797/ using document root <none> and password file <none>
I20260812 16:45:29.955183 10566 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:29.955309 10566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:29.955667 10566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:29.959088 10566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "451c0533d4f14b9aa1c69bc314d00a36"
format_stamp: "Formatted at 2026-08-12 16:45:29 on dist-test-slave-1vmg"
server_key: "b01a281e1b908bf39a83b7a55845ee9b"
server_key_iv: "98552ec934668e2db6ca58b51bf1308e"
server_key_version: "encryptionkey@0"
I20260812 16:45:29.960073 10566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "451c0533d4f14b9aa1c69bc314d00a36"
format_stamp: "Formatted at 2026-08-12 16:45:29 on dist-test-slave-1vmg"
server_key: "b01a281e1b908bf39a83b7a55845ee9b"
server_key_iv: "98552ec934668e2db6ca58b51bf1308e"
server_key_version: "encryptionkey@0"
I20260812 16:45:29.966573 10566 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260812 16:45:29.972373 10589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:29.974797 10566 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:45:29.975142 10566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "451c0533d4f14b9aa1c69bc314d00a36"
format_stamp: "Formatted at 2026-08-12 16:45:29 on dist-test-slave-1vmg"
server_key: "b01a281e1b908bf39a83b7a55845ee9b"
server_key_iv: "98552ec934668e2db6ca58b51bf1308e"
server_key_version: "encryptionkey@0"
I20260812 16:45:29.975484 10566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:30.023648 10566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:30.029829 10566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:30.030148 10566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:30.033962 10566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:30.034081 10566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:30.034258 10566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:30.034344 10566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:45:30 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:30.063012 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.035312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44497 (local address 127.28.243.62:44101)
0812 16:45:30.038340 (+ 3028us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:30.038347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:30.038375 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:45:30.040549 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:30.040557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:30.040686 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:30.040832 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:30.043471 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.044681 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.047477 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.048045 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.048179 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.055188 (+ 7009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:30.055197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:30.055201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:30.055264 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:30.059009 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.060371 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.060377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.060381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.060566 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.061062 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.061067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.061070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.061379 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:30.061524 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:30.062334 (+ 810us) server_negotiation.cc:299] Negotiation successful
0812 16:45:30.062595 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2877,"thread_start_us":2759,"threads_started":1}
I20260812 16:45:30.064402 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.036094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.28.243.1:44497)
0812 16:45:30.037436 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:30.037531 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:30.040149 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:30.041215 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:30.041229 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:30.042105 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:30.043180 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.043205 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.044900 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.044906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.047231 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.047258 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.048548 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.048556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.048814 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.049641 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:30.049694 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:30.054623 (+ 4929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:30.059215 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.059228 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.059255 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.060183 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.060690 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.060696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.060700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.060930 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.061524 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:30.061533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:30.061931 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 16:45:30.062197 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 16:45:30.062789 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":542,"threads_started":1}
I20260812 16:45:30.087672 10566 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:35919
I20260812 16:45:30.087687 10778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:35919 every 8 connection(s)
I20260812 16:45:30.089452 10566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:30.091966 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 10566
I20260812 16:45:30.092191 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:45:30.092581 10780 heartbeater.cc:347] Connected to a master server at 127.28.243.62:44101
I20260812 16:45:30.092898 10780 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:30.093029 29644 external_mini_cluster.cc:1428] Setting key 9a30023431baa1d9b0a99d8f726fc4b1
I20260812 16:45:30.093905 10780 heartbeater.cc:511] Master 127.28.243.62:44101 requested a full tablet report, sending...
I20260812 16:45:30.096028 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.077802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:57009 (local address 127.28.243.62:44101)
0812 16:45:30.078060 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:30.078069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:30.078746 (+ 677us) server_negotiation.cc:407] Connection header received
0812 16:45:30.079017 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:30.079021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:30.079106 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:30.079217 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:30.081954 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.083208 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.085607 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.086156 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.086311 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.088972 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:30.088977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:30.088981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:30.089020 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:30.092864 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.093610 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.093616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.093619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.093789 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.094282 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.094286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.094289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.094678 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:30.095212 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:30.095375 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 16:45:30.095631 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":85}
I20260812 16:45:30.096542 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.077653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.28.243.1:57009)
0812 16:45:30.078556 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:30.078582 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:30.078801 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:30.079288 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:30.079293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:30.081131 (+ 1838us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:30.081599 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.081617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.083430 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.083441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.085392 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.085414 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.086436 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.086444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.086661 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.087514 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:30.087541 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:30.088705 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:30.093018 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.093024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.093029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.093480 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.093953 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.093959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.093962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.094139 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.094817 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:30.094822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:30.095064 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 16:45:30.095226 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 16:45:30.095868 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":519,"threads_started":1}
I20260812 16:45:30.096606 10434 ts_manager.cc:194] Registered new tserver with Master: 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919)
I20260812 16:45:30.098764 10434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:44497
WARNING: no policy specified for oryx/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab.
Entry for principal oryx/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab.
I20260812 16:45:30.163777 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44101
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:30.536512 10786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:30.536836 10786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:30.536969 10786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:30.544643 10786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:30.544795 10786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:30.544914 10786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:45:30.545044 10786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:30.555692 10786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:44101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:30.557808 10786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:30.559837 10786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:30.576850 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:30.577014 10800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:30.576790 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:30.579101 10786 server_base.cc:1034] running on GCE node
I20260812 16:45:30.579947 10786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:30.581362 10786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:30.582823 10786 hybrid_clock.cc:648] HybridClock initialized: now 1786553130582706 us; error 98 us; skew 500 ppm
I20260812 16:45:30.583227 10786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:30 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:30.587011 10786 init.cc:377] Logged in from keytab as oryx/127.28.243.2@KRBTEST.COM (short username oryx)
I20260812 16:45:30.588675 10786 webserver.cc:460] Webserver started at http://127.28.243.2:44097/ using document root <none> and password file <none>
I20260812 16:45:30.589452 10786 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:30.589564 10786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:30.589920 10786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:30.593350 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "bdec7c95156142049685c77145f9a067"
format_stamp: "Formatted at 2026-08-12 16:45:30 on dist-test-slave-1vmg"
server_key: "6d4687bb49d568f8487242a939223a2c"
server_key_iv: "81e766c9cb88fbfa7ffed90ceccc0d8d"
server_key_version: "encryptionkey@0"
I20260812 16:45:30.594245 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "bdec7c95156142049685c77145f9a067"
format_stamp: "Formatted at 2026-08-12 16:45:30 on dist-test-slave-1vmg"
server_key: "6d4687bb49d568f8487242a939223a2c"
server_key_iv: "81e766c9cb88fbfa7ffed90ceccc0d8d"
server_key_version: "encryptionkey@0"
I20260812 16:45:30.600766 10786 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:45:30.605121 10811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:30.606796 10786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:30.607128 10786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "bdec7c95156142049685c77145f9a067"
format_stamp: "Formatted at 2026-08-12 16:45:30 on dist-test-slave-1vmg"
server_key: "6d4687bb49d568f8487242a939223a2c"
server_key_iv: "81e766c9cb88fbfa7ffed90ceccc0d8d"
server_key_version: "encryptionkey@0"
I20260812 16:45:30.607409 10786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:30.629940 10786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:30.635567 10786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:30.635890 10786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:30.639082 10786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:30.639207 10786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:30.639372 10786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:30.639453 10786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:45:30 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553130, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:30.669978 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.639653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:43989 (local address 127.28.243.62:44101)
0812 16:45:30.640528 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:30.640533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:30.641707 (+ 1174us) server_negotiation.cc:407] Connection header received
0812 16:45:30.644079 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:30.644085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:30.644191 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:30.644296 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:30.646623 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.647612 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.650881 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.651877 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.652109 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.659545 (+ 7436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:30.659555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:30.659561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:30.659627 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:30.663865 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.665444 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.665452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.665455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.665696 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.666495 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.666501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.666505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.667128 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:30.667781 (+ 653us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:30.669172 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0812 16:45:30.669532 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":667,"threads_started":1}
I20260812 16:45:30.673192 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.640514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.28.243.2:43989)
0812 16:45:30.641488 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:30.641540 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:30.643743 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:30.644616 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:30.644629 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:30.645526 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:30.646379 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.646415 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.647851 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.647857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.650644 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.650667 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.652345 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.652356 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.652811 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.653980 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:30.654014 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:30.658974 (+ 4960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:30.664158 (+ 5184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.664172 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.664195 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.665171 (+ 976us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.665891 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.665897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.665906 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.666309 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.667642 (+ 1333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:30.667656 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:30.668534 (+ 878us) client_negotiation.cc:769] Sending connection context
0812 16:45:30.670483 (+ 1949us) client_negotiation.cc:240] Negotiation successful
0812 16:45:30.671337 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":442,"threads_started":1}
I20260812 16:45:30.682760 10786 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:38401
I20260812 16:45:30.682799 11001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:38401 every 8 connection(s)
I20260812 16:45:30.684497 10786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:30.687146 11002 heartbeater.cc:347] Connected to a master server at 127.28.243.62:44101
I20260812 16:45:30.687410 11002 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:30.688105 11002 heartbeater.cc:511] Master 127.28.243.62:44101 requested a full tablet report, sending...
I20260812 16:45:30.689625 10434 ts_manager.cc:194] Registered new tserver with Master: bdec7c95156142049685c77145f9a067 (127.28.243.2:38401)
I20260812 16:45:30.690490 10434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:43989
I20260812 16:45:30.691107 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 10786
I20260812 16:45:30.691337 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:45:30.692139 29644 external_mini_cluster.cc:1428] Setting key 476cad9163ff42d26258688313081006
I20260812 16:45:30.700809 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.681655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:46829 (local address 127.28.243.62:44101)
0812 16:45:30.681917 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:30.681926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:30.682683 (+ 757us) server_negotiation.cc:407] Connection header received
0812 16:45:30.682890 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:30.682895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:30.683015 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:30.683116 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:30.684181 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.684970 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.687446 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.688048 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.688594 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.692386 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:30.692393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:30.692398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:30.692679 (+ 281us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:30.696674 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.697874 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.697879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.697882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.698073 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:30.698626 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:30.698632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:30.698635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:30.699171 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:30.699374 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:30.700297 (+ 923us) server_negotiation.cc:299] Negotiation successful
0812 16:45:30.700555 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 16:45:30.701006 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:30.681809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.28.243.2:46829)
0812 16:45:30.682547 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:30.682569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:30.682756 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:30.683176 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:30.683180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:30.683695 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:30.684008 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.684019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.685132 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.685137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.687261 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:30.687279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:30.688664 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:30.688681 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:30.689113 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:30.690083 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:30.690109 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:30.692095 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:30.696955 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.696963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.696967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.697657 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.698211 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:30.698217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:30.698220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:30.698415 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:30.699444 (+ 1029us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:30.699455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:30.699905 (+ 450us) client_negotiation.cc:769] Sending connection context
0812 16:45:30.700142 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 16:45:30.700587 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":403,"threads_started":1}
WARNING: no policy specified for oryx/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab.
Entry for principal oryx/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab.
I20260812 16:45:30.768898 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:44101
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:31.102263 10780 heartbeater.cc:503] Master 127.28.243.62:44101 was elected leader, sending a full tablet report...
W20260812 16:45:31.146184 11007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:31.146656 11007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:31.146858 11007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:31.154755 11007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:31.154906 11007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:31.155054 11007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:45:31.155175 11007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:31.167415 11007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.62:44101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:31.169061 11007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:31.170650 11007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:31.187132 11017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:31.187534 11012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:31.187158 11014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:31.189039 11007 server_base.cc:1034] running on GCE node
I20260812 16:45:31.189810 11007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:31.191104 11007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:31.192391 11007 hybrid_clock.cc:648] HybridClock initialized: now 1786553131192312 us; error 51 us; skew 500 ppm
I20260812 16:45:31.192770 11007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:31 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:31.197417 11007 init.cc:377] Logged in from keytab as oryx/127.28.243.3@KRBTEST.COM (short username oryx)
I20260812 16:45:31.199859 11007 webserver.cc:460] Webserver started at http://127.28.243.3:44951/ using document root <none> and password file <none>
I20260812 16:45:31.200978 11007 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:31.201128 11007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:31.201651 11007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:31.205725 11007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "d3d078536c7641c0b09afd075c329b7d"
format_stamp: "Formatted at 2026-08-12 16:45:31 on dist-test-slave-1vmg"
server_key: "a682ee9bf7d075170073e8664469a616"
server_key_iv: "3df1cc0c4ccec1e678099e00ba614e7e"
server_key_version: "encryptionkey@0"
I20260812 16:45:31.206759 11007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d078536c7641c0b09afd075c329b7d"
format_stamp: "Formatted at 2026-08-12 16:45:31 on dist-test-slave-1vmg"
server_key: "a682ee9bf7d075170073e8664469a616"
server_key_iv: "3df1cc0c4ccec1e678099e00ba614e7e"
server_key_version: "encryptionkey@0"
I20260812 16:45:31.214869 11007 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s
I20260812 16:45:31.219545 11031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:31.221643 11007 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:45:31.221856 11007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "d3d078536c7641c0b09afd075c329b7d"
format_stamp: "Formatted at 2026-08-12 16:45:31 on dist-test-slave-1vmg"
server_key: "a682ee9bf7d075170073e8664469a616"
server_key_iv: "3df1cc0c4ccec1e678099e00ba614e7e"
server_key_version: "encryptionkey@0"
I20260812 16:45:31.222101 11007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:31.237075 11007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:31.243255 11007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:31.243573 11007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:31.247577 11007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:31.247709 11007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:31.247889 11007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:31.248049 11007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:45:31 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553131, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:31.280720 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:31.248771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.28.243.3:58425)
0812 16:45:31.250067 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:31.250139 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:31.253543 (+ 3404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:31.255119 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:31.255136 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:31.256430 (+ 1294us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:31.257545 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:31.257581 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.259786 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.259794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:31.262231 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:31.262265 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.263647 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.263657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:31.263902 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:31.264817 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:31.264858 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:31.268432 (+ 3574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:31.273213 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:31.273227 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:31.273254 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:31.274100 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:31.274978 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:31.274985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:31.274989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:31.275431 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:31.276416 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:31.276429 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:31.277209 (+ 780us) client_negotiation.cc:769] Sending connection context
0812 16:45:31.277650 (+ 441us) client_negotiation.cc:240] Negotiation successful
0812 16:45:31.278449 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":523,"threads_started":1}
I20260812 16:45:31.281836 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:31.247551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:58425 (local address 127.28.243.62:44101)
0812 16:45:31.248305 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:31.248313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:31.250327 (+ 2014us) server_negotiation.cc:407] Connection header received
0812 16:45:31.254079 (+ 3752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:31.254088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:31.254220 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:31.254309 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:31.257909 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.259398 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.262621 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.263339 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.263484 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:31.268814 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:31.268822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:31.268828 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:31.268948 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:31.272981 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:31.274418 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:31.274425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:31.274428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:31.274701 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:31.275685 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:31.275693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:31.275697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:31.276199 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:31.280636 (+ 4437us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:31.280980 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 16:45:31.281341 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":472,"threads_started":1}
I20260812 16:45:31.306222 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:31.289511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:57773 (local address 127.28.243.62:44101)
0812 16:45:31.289961 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:31.289970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:31.290623 (+ 653us) server_negotiation.cc:407] Connection header received
0812 16:45:31.290957 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:31.290964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:31.291055 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:31.291163 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:31.293455 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.294424 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.297099 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.297617 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.297737 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:31.300485 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:31.300490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:31.300493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:31.300530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:31.303405 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:31.304170 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:31.304175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:31.304179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:31.304315 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:31.304676 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:31.304681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:31.304685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:31.305024 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:31.305160 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:31.305612 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 16:45:31.305847 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260812 16:45:31.306314 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:31.289418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.28.243.3:57773)
0812 16:45:31.290453 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:31.290474 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:31.290750 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:31.291214 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:31.291219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:31.292477 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:31.293140 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:31.293162 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.294672 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.294688 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:31.296809 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:31.296838 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.298067 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.298074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:31.298305 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:31.299183 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:31.299206 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:31.300247 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:31.303557 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:31.303564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:31.303569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:31.304036 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:31.304429 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:31.304435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:31.304440 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:31.304570 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:31.305150 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:31.305155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:31.305346 (+ 191us) client_negotiation.cc:769] Sending connection context
0812 16:45:31.305487 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 16:45:31.305853 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":661,"threads_started":1}
I20260812 16:45:31.317929 11007 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:34815
I20260812 16:45:31.319717 11007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:31.317950 11227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:34815 every 8 connection(s)
I20260812 16:45:31.325500 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 11007
I20260812 16:45:31.325744 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:45:31.326335 29644 external_mini_cluster.cc:1428] Setting key 8ca8c4b1ddfa5f3d2a59c24c6e438c3c
I20260812 16:45:31.327364 11230 heartbeater.cc:347] Connected to a master server at 127.28.243.62:44101
I20260812 16:45:31.327804 11230 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:31.329003 11230 heartbeater.cc:511] Master 127.28.243.62:44101 requested a full tablet report, sending...
I20260812 16:45:31.331090 10434 ts_manager.cc:194] Registered new tserver with Master: d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:31.332317 10434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:58425
I20260812 16:45:31.343214 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:45:31 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553131, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:45:31 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:31 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:31.693728 11002 heartbeater.cc:503] Master 127.28.243.62:44101 was elected leader, sending a full tablet report...
I20260812 16:45:32.336239 11230 heartbeater.cc:503] Master 127.28.243.62:44101 was elected leader, sending a full tablet report...
W20260812 16:45:32.483783 11243 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.069s user 0.010s sys 0.015s
W20260812 16:45:32.485107 11044 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:31.406877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.28.243.62:44101)
0812 16:45:31.407168 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:31.407174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:31.407790 (+ 616us) server_negotiation.cc:407] Connection header received
0812 16:45:31.408145 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:31.408152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:31.408244 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:31.408358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:31.410239 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.411133 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.412497 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:31.412931 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:31.413350 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.484831 (+1071481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33730: BlockingRecv error: recv got EOF from 127.0.0.1:33730 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 12 16:45:32 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553131, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:32.506392 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.489556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.28.243.62:44101)
0812 16:45:32.489808 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.489816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.490274 (+ 458us) server_negotiation.cc:407] Connection header received
0812 16:45:32.490525 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.490529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.490604 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.490705 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:32.492690 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.493516 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.494972 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.495462 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.495587 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.500304 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:32.500310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:32.500313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:32.500361 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:32.503563 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:32.504374 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:32.504378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:32.504381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:32.504520 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:32.504910 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:32.504913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:32.504916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:32.505211 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:32.505449 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.505831 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.506083 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 16:45:32.511457 10434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:32.518602 10434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:32.543275 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.533119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.28.243.3:34815)
0812 16:45:32.533829 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.533837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.533868 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:45:32.534193 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.534201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.534459 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.534648 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.535675 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.537575 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.540629 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.542017 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.542163 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.542421 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.542731 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.542945 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":771,"threads_started":1}
I20260812 16:45:32.543409 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.532718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:34815 (local address 127.0.0.1:47298)
0812 16:45:32.533746 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.533784 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.533965 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.534899 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.534906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.534970 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.535440 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.535453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.537773 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.537779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.540397 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.540414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.542246 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.542253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.542497 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.542545 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.542669 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.542985 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":46,"thread_start_us":540,"threads_started":1}
I20260812 16:45:32.544364 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.532418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.28.243.1:35919)
0812 16:45:32.533003 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.533012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.534231 (+ 1219us) server_negotiation.cc:407] Connection header received
0812 16:45:32.534499 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.534506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.534734 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.534893 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.535853 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.537575 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.541306 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.542794 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.542949 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.543159 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.543912 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.544094 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":339,"threads_started":1}
I20260812 16:45:32.544373 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.532258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35919 (local address 127.0.0.1:60284)
0812 16:45:32.533586 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.534157 (+ 571us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.534327 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.535167 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.535171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.535196 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.535654 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.535666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.537789 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.537793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.541089 (+ 3296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.541106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.543527 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.543536 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.543708 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.543750 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.543841 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.543980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"spinlock_wait_cycles":6400,"thread_start_us":1054,"threads_started":1}
I20260812 16:45:32.547441 11116 tablet_service.cc:1467] Processing CreateTablet for tablet acb91775e087498f95fb63140396541d (DEFAULT_TABLE table=test-table [id=5c94f3fecfe745508dd3f99b579cc84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:32.547788 10665 tablet_service.cc:1467] Processing CreateTablet for tablet acb91775e087498f95fb63140396541d (DEFAULT_TABLE table=test-table [id=5c94f3fecfe745508dd3f99b579cc84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:32.549160 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb91775e087498f95fb63140396541d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:32.549207 11116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb91775e087498f95fb63140396541d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:32.549501 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.534359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.28.243.2:38401)
0812 16:45:32.539077 (+ 4718us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.539099 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.539132 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:45:32.539247 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.539256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.539532 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.539773 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.540573 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.542043 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.545290 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.548237 (+ 2947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.548408 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.548625 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.549019 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.549197 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4501,"thread_start_us":4358,"threads_started":1}
I20260812 16:45:32.549767 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.534084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:38401 (local address 127.0.0.1:41264)
0812 16:45:32.538349 (+ 4265us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.538369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.538496 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.539775 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.539780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.539813 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.540328 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.540343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.542230 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.542236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.545079 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.545095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.548606 (+ 3511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.548613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.548804 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.548842 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.548950 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.549096 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4109,"mutex_wait_us":48,"thread_start_us":3940,"threads_started":1}
I20260812 16:45:32.560050 10889 tablet_service.cc:1467] Processing CreateTablet for tablet acb91775e087498f95fb63140396541d (DEFAULT_TABLE table=test-table [id=5c94f3fecfe745508dd3f99b579cc84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:32.561925 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb91775e087498f95fb63140396541d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:32.563663 11279 tablet_bootstrap.cc:492] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Bootstrap starting.
I20260812 16:45:32.569461 11279 tablet_bootstrap.cc:654] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:32.570755 11280 tablet_bootstrap.cc:492] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Bootstrap starting.
I20260812 16:45:32.571079 11279 log.cc:824] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:32.576916 11279 tablet_bootstrap.cc:492] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: No bootstrap required, opened a new log
I20260812 16:45:32.577602 11279 ts_tablet_manager.cc:1397] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20260812 16:45:32.579289 11280 tablet_bootstrap.cc:654] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:32.580878 11280 log.cc:824] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:32.582634 11282 tablet_bootstrap.cc:492] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Bootstrap starting.
I20260812 16:45:32.585115 11280 tablet_bootstrap.cc:492] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: No bootstrap required, opened a new log
I20260812 16:45:32.585470 11280 ts_tablet_manager.cc:1397] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20260812 16:45:32.587561 11282 tablet_bootstrap.cc:654] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:32.588649 11282 log.cc:824] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:32.590425 11279 raft_consensus.cc:348] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.591493 11279 raft_consensus.cc:374] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:32.591642 11279 raft_consensus.cc:729] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d078536c7641c0b09afd075c329b7d, State: Initialized, Role: FOLLOWER
I20260812 16:45:32.591740 11282 tablet_bootstrap.cc:492] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: No bootstrap required, opened a new log
I20260812 16:45:32.592059 11282 ts_tablet_manager.cc:1397] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 16:45:32.593025 11279 consensus_queue.cc:260] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.599779 11280 raft_consensus.cc:348] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.600157 11282 raft_consensus.cc:348] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.600534 11280 raft_consensus.cc:374] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:32.600648 11282 raft_consensus.cc:374] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:32.600664 11280 raft_consensus.cc:729] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c0533d4f14b9aa1c69bc314d00a36, State: Initialized, Role: FOLLOWER
I20260812 16:45:32.600759 11282 raft_consensus.cc:729] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdec7c95156142049685c77145f9a067, State: Initialized, Role: FOLLOWER
I20260812 16:45:32.601562 11282 consensus_queue.cc:260] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.601579 11280 consensus_queue.cc:260] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.603529 11282 ts_tablet_manager.cc:1428] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260812 16:45:32.605847 11279 ts_tablet_manager.cc:1428] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Time spent starting tablet: real 0.028s user 0.020s sys 0.004s
I20260812 16:45:32.607623 11280 ts_tablet_manager.cc:1428] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Time spent starting tablet: real 0.022s user 0.016s sys 0.006s
I20260812 16:45:32.654104 11288 raft_consensus.cc:482] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:32.654481 11288 raft_consensus.cc:504] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.656423 11288 leader_election.cc:288] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:32.662235 11287 raft_consensus.cc:482] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:32.662580 11287 raft_consensus.cc:504] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.663643 11289 raft_consensus.cc:482] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:32.663997 11289 raft_consensus.cc:504] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.665288 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.657076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:44269 (local address 127.28.243.1:35919)
0812 16:45:32.657347 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.657353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.657845 (+ 492us) server_negotiation.cc:407] Connection header received
0812 16:45:32.658087 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.658091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.658161 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.658253 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.658889 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.660068 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.662632 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.663857 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.664051 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.664135 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.664837 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.665018 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 16:45:32.665947 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.657413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:39413 (local address 127.28.243.3:34815)
0812 16:45:32.657652 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.657659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.658174 (+ 515us) server_negotiation.cc:407] Connection header received
0812 16:45:32.658378 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.658384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.658456 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.658544 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.659161 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.660248 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.663284 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.664646 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.664805 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.664880 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.665527 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.665691 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 16:45:32.665335 11287 leader_election.cc:288] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), bdec7c95156142049685c77145f9a067 (127.28.243.2:38401)
I20260812 16:45:32.666271 11289 leader_election.cc:288] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815), bdec7c95156142049685c77145f9a067 (127.28.243.2:38401)
I20260812 16:45:32.666834 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.657145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:34815 (local address 127.28.243.2:39413)
0812 16:45:32.658065 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.658086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.658200 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.658584 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.658593 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.658616 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.659012 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.659023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.660794 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.660800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.662800 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.662815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.665116 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.665123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.665303 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.665341 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.666132 (+ 791us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.666327 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":614,"threads_started":1}
I20260812 16:45:32.668180 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d" is_pre_election: true
I20260812 16:45:32.668823 11158 raft_consensus.cc:2454] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:32.669714 10817 leader_election.cc:302] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdec7c95156142049685c77145f9a067, d3d078536c7641c0b09afd075c329b7d; no voters:
I20260812 16:45:32.670682 11288 raft_consensus.cc:2781] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:32.670871 11288 raft_consensus.cc:482] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:32.670989 11288 raft_consensus.cc:3037] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:32.674638 11288 raft_consensus.cc:504] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.676422 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d"
I20260812 16:45:32.676702 11158 raft_consensus.cc:3037] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:32.678906 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.665796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:38401 (local address 127.28.243.3:41115)
0812 16:45:32.668946 (+ 3150us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.668972 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.669175 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.670505 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.670511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.670547 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.671055 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.671069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.673311 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.673317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.675836 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.675855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.677890 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.677897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.678097 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.678133 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.678232 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.678409 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2660,"mutex_wait_us":1,"thread_start_us":6422,"threads_started":1}
I20260812 16:45:32.679728 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.666822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:41115 (local address 127.28.243.2:38401)
0812 16:45:32.667105 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.667112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.670098 (+ 2986us) server_negotiation.cc:407] Connection header received
0812 16:45:32.670179 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.670185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.670268 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.670357 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.671927 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.673095 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.676062 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.677325 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.677523 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.677603 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.679300 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.679466 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1358,"server-negotiator.queue_time_us":76}
I20260812 16:45:32.680729 11288 leader_election.cc:288] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Requested vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:32.680527 11158 raft_consensus.cc:2454] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:32.681965 10817 leader_election.cc:302] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdec7c95156142049685c77145f9a067, d3d078536c7641c0b09afd075c329b7d; no voters:
I20260812 16:45:32.682200 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.672498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:41943 (local address 127.28.243.3:34815)
0812 16:45:32.672640 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.672645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.672661 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:45:32.672713 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.672717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.672777 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.672858 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.673791 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.675238 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.678629 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.680199 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.680410 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.680494 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.681751 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.681917 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 16:45:32.682469 11288 raft_consensus.cc:2781] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:32.682612 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.666962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:34815 (local address 127.28.243.1:41943)
0812 16:45:32.669651 (+ 2689us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.669671 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.669828 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.673021 (+ 3193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.673027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.673073 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.673584 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.673598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.675475 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.675482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.678348 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.678375 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.680777 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.680785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.681115 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.681196 (+ 81us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.681315 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.681482 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2538,"mutex_wait_us":47,"spinlock_wait_cycles":95488,"thread_start_us":1722,"threads_started":1}
I20260812 16:45:32.684105 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.656889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35919 (local address 127.28.243.2:44269)
0812 16:45:32.657686 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.657709 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.657928 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.658294 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.658299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.658325 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.658691 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.658709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.660265 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.660271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.662462 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.662482 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.664317 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.664324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.664510 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.664548 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.669266 (+ 4718us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.683664 (+ 14398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":4,"thread_start_us":467,"threads_started":1}
I20260812 16:45:32.685575 10692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36" is_pre_election: true
I20260812 16:45:32.685860 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "451c0533d4f14b9aa1c69bc314d00a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d" is_pre_election: true
I20260812 16:45:32.686273 10692 raft_consensus.cc:2454] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:32.686342 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36"
I20260812 16:45:32.686813 11158 raft_consensus.cc:2379] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 451c0533d4f14b9aa1c69bc314d00a36 in current term 1: Already voted for candidate bdec7c95156142049685c77145f9a067 in this term.
I20260812 16:45:32.686822 10691 raft_consensus.cc:3037] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Advancing to term 1
W20260812 16:45:32.688807 11003 tablet.cc:2369] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:32.690615 10691 raft_consensus.cc:2454] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:32.693748 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.670299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:58453 (local address 127.28.243.2:38401)
0812 16:45:32.680163 (+ 9864us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.680168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.680189 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:45:32.680264 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.680269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.680352 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.680453 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.681563 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.683380 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.686747 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.688224 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.692985 (+ 4761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.693089 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.693163 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.693356 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9752,"threads_started":1}
I20260812 16:45:32.694171 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.671682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:35529 (local address 127.28.243.1:35919)
0812 16:45:32.672871 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.672878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.672894 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:45:32.672953 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.672957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.673024 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.673099 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:45:32.683921 (+ 10822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.685104 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.691411 (+ 6307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.692740 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.692915 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.692995 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.693657 (+ 662us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.693801 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":1067}
I20260812 16:45:32.694546 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.666033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35919 (local address 127.28.243.3:35529)
0812 16:45:32.672482 (+ 6449us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.672507 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.672625 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.678673 (+ 6048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.678679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.678730 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.683702 (+ 4972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.683717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.685346 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.685352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.690487 (+ 5135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.690515 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.693243 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.693252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.693464 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.693502 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.693974 (+ 472us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.694140 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6325,"mutex_wait_us":4492,"thread_start_us":2841,"threads_started":1}
I20260812 16:45:32.694787 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.669653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:38401 (local address 127.28.243.1:58453)
0812 16:45:32.671781 (+ 2128us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:32.671803 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:32.671941 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:32.680673 (+ 8732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:32.680678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:32.680724 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:45:32.681279 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.681296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.683530 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.683536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.686229 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:32.686270 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.690399 (+ 4129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.690408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:32.690612 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.690656 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 16:45:32.690748 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 16:45:32.690904 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1940,"mutex_wait_us":76,"thread_start_us":1676,"threads_started":1}
I20260812 16:45:32.695325 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "d3d078536c7641c0b09afd075c329b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36" is_pre_election: true
I20260812 16:45:32.695781 10691 raft_consensus.cc:2379] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3d078536c7641c0b09afd075c329b7d in current term 1: Already voted for candidate bdec7c95156142049685c77145f9a067 in this term.
I20260812 16:45:32.699391 11288 raft_consensus.cc:686] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Becoming Leader. State: Replica: bdec7c95156142049685c77145f9a067, State: Running, Role: LEADER
I20260812 16:45:32.700107 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "451c0533d4f14b9aa1c69bc314d00a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdec7c95156142049685c77145f9a067" is_pre_election: true
I20260812 16:45:32.700115 11288 consensus_queue.cc:237] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } }
I20260812 16:45:32.700675 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb91775e087498f95fb63140396541d" candidate_uuid: "d3d078536c7641c0b09afd075c329b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdec7c95156142049685c77145f9a067" is_pre_election: true
I20260812 16:45:32.701892 10591 leader_election.cc:302] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36; no voters: bdec7c95156142049685c77145f9a067, d3d078536c7641c0b09afd075c329b7d
I20260812 16:45:32.702096 11033 leader_election.cc:302] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3d078536c7641c0b09afd075c329b7d; no voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067
I20260812 16:45:32.702508 11289 raft_consensus.cc:2726] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 16:45:32.702800 11287 raft_consensus.cc:2726] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 16:45:32.707127 10433 catalog_manager.cc:5495] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdec7c95156142049685c77145f9a067 (127.28.243.2). New cstate: current_term: 1 leader_uuid: "bdec7c95156142049685c77145f9a067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } health_report { overall_health: HEALTHY } } }
I20260812 16:45:32.746693 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:32.738878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.28.243.2:38401)
0812 16:45:32.739212 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:32.739217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:32.739249 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:45:32.739397 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:32.739401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:32.739554 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:32.739892 (+ 338us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:32.740519 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.741420 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.743025 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:32.743597 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:32.743710 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:32.744495 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:32.745593 (+ 1098us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:32.745762 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:32.746067 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 16:45:32.746190 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260812 16:45:32.756275 11158 raft_consensus.cc:1261] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:32.756289 10691 raft_consensus.cc:1261] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:32.757755 11301 consensus_queue.cc:1035] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:32.758188 11288 consensus_queue.cc:1035] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:32.772665 11310 mvcc.cc:205] Tried to move back new op lower bound from 7317721631758000128 to 7317721631546400768. Current Snapshot: MvccSnapshot[applied={T|T < 7317721631758000128}]
I20260812 16:45:32.787225 10434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:32.787528 10434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:32.791829 10434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Sending DeleteTablet for 3 replicas of tablet acb91775e087498f95fb63140396541d
I20260812 16:45:32.793275 10665 tablet_service.cc:1514] Processing DeleteTablet for tablet acb91775e087498f95fb63140396541d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:32 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60284
I20260812 16:45:32.793608 11116 tablet_service.cc:1514] Processing DeleteTablet for tablet acb91775e087498f95fb63140396541d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:32 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:47298
I20260812 16:45:32.794399 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet acb91775e087498f95fb63140396541d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:32 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:41264
I20260812 16:45:32.794659 10434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:32.795467 10434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:32.805351 10663 tablet_service.cc:1467] Processing CreateTablet for tablet e8c69f8209c8454e8fe90e3eab52d930 (DEFAULT_TABLE table=test-table [id=f0d0effc270f4aca8a315e014bc9d639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:32.805588 10888 tablet_service.cc:1467] Processing CreateTablet for tablet e8c69f8209c8454e8fe90e3eab52d930 (DEFAULT_TABLE table=test-table [id=f0d0effc270f4aca8a315e014bc9d639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:32.805998 11115 tablet_service.cc:1467] Processing CreateTablet for tablet e8c69f8209c8454e8fe90e3eab52d930 (DEFAULT_TABLE table=test-table [id=f0d0effc270f4aca8a315e014bc9d639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:32.806162 10663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c69f8209c8454e8fe90e3eab52d930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:32.806288 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c69f8209c8454e8fe90e3eab52d930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:32.806701 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c69f8209c8454e8fe90e3eab52d930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:32.808379 11319 tablet_replica.cc:330] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: stopping tablet replica
I20260812 16:45:32.810884 11321 tablet_replica.cc:330] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: stopping tablet replica
I20260812 16:45:32.811403 11321 raft_consensus.cc:2229] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:32.808907 11319 raft_consensus.cc:2229] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:32.813215 11280 tablet_bootstrap.cc:492] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Bootstrap starting.
I20260812 16:45:32.813886 11282 tablet_bootstrap.cc:492] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Bootstrap starting.
I20260812 16:45:32.813860 11321 raft_consensus.cc:2258] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:32.814602 11319 raft_consensus.cc:2258] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:32.815495 11280 tablet_bootstrap.cc:654] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:32.815687 11282 tablet_bootstrap.cc:654] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:32.819100 11279 tablet_bootstrap.cc:492] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Bootstrap starting.
I20260812 16:45:32.821061 11279 tablet_bootstrap.cc:654] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:32.826534 11280 tablet_bootstrap.cc:492] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: No bootstrap required, opened a new log
I20260812 16:45:32.826743 11280 ts_tablet_manager.cc:1397] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260812 16:45:32.828018 11280 raft_consensus.cc:348] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.828293 11280 raft_consensus.cc:374] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:32.828395 11280 raft_consensus.cc:729] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c0533d4f14b9aa1c69bc314d00a36, State: Initialized, Role: FOLLOWER
I20260812 16:45:32.828706 11280 consensus_queue.cc:260] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.818784 11319 ts_tablet_manager.cc:1905] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:32.829658 11282 tablet_bootstrap.cc:492] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: No bootstrap required, opened a new log
I20260812 16:45:32.830008 11282 ts_tablet_manager.cc:1397] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20260812 16:45:32.830189 11280 ts_tablet_manager.cc:1428] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:32.830861 11318 tablet_replica.cc:330] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: stopping tablet replica
I20260812 16:45:32.831238 11282 raft_consensus.cc:348] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.831810 11282 raft_consensus.cc:374] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:32.831872 11279 tablet_bootstrap.cc:492] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: No bootstrap required, opened a new log
I20260812 16:45:32.832023 11282 raft_consensus.cc:729] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdec7c95156142049685c77145f9a067, State: Initialized, Role: FOLLOWER
I20260812 16:45:32.832068 11279 ts_tablet_manager.cc:1397] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260812 16:45:32.831621 11321 ts_tablet_manager.cc:1905] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:32.833014 11279 raft_consensus.cc:348] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.833266 11279 raft_consensus.cc:374] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:32.833374 11279 raft_consensus.cc:729] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d078536c7641c0b09afd075c329b7d, State: Initialized, Role: FOLLOWER
I20260812 16:45:32.833474 11318 raft_consensus.cc:2229] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:32.833554 11282 consensus_queue.cc:260] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.833671 11279 consensus_queue.cc:260] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.834069 11318 raft_consensus.cc:2258] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:32.835275 11279 ts_tablet_manager.cc:1428] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 16:45:32.835258 11282 ts_tablet_manager.cc:1428] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 16:45:32.836473 11318 ts_tablet_manager.cc:1905] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:32.842646 11321 ts_tablet_manager.cc:1918] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:32.843128 11321 log.cc:1196] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/acb91775e087498f95fb63140396541d
I20260812 16:45:32.843683 11318 ts_tablet_manager.cc:1918] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:32.843976 11321 ts_tablet_manager.cc:1939] T acb91775e087498f95fb63140396541d P bdec7c95156142049685c77145f9a067: Deleting consensus metadata
W20260812 16:45:32.844137 10781 tablet.cc:2369] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:32.844888 11318 log.cc:1196] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/acb91775e087498f95fb63140396541d
I20260812 16:45:32.845996 11318 ts_tablet_manager.cc:1939] T acb91775e087498f95fb63140396541d P 451c0533d4f14b9aa1c69bc314d00a36: Deleting consensus metadata
I20260812 16:45:32.846470 10407 catalog_manager.cc:4841] TS bdec7c95156142049685c77145f9a067 (127.28.243.2:38401): tablet acb91775e087498f95fb63140396541d (table test-table [id=5c94f3fecfe745508dd3f99b579cc84a]) successfully deleted
I20260812 16:45:32.848315 10406 catalog_manager.cc:4841] TS 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919): tablet acb91775e087498f95fb63140396541d (table test-table [id=5c94f3fecfe745508dd3f99b579cc84a]) successfully deleted
I20260812 16:45:32.850587 11319 ts_tablet_manager.cc:1918] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:32.850780 11319 log.cc:1196] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/acb91775e087498f95fb63140396541d
I20260812 16:45:32.851284 11319 ts_tablet_manager.cc:1939] T acb91775e087498f95fb63140396541d P d3d078536c7641c0b09afd075c329b7d: Deleting consensus metadata
I20260812 16:45:32.852891 10407 catalog_manager.cc:4841] TS d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815): tablet acb91775e087498f95fb63140396541d (table test-table [id=5c94f3fecfe745508dd3f99b579cc84a]) successfully deleted
I20260812 16:45:32.953195 11313 raft_consensus.cc:482] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:32.953625 11313 raft_consensus.cc:504] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.954838 11313 leader_election.cc:288] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:32.955845 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c69f8209c8454e8fe90e3eab52d930" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36" is_pre_election: true
I20260812 16:45:32.955891 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c69f8209c8454e8fe90e3eab52d930" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d" is_pre_election: true
I20260812 16:45:32.956174 10691 raft_consensus.cc:2454] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:32.956247 11158 raft_consensus.cc:2454] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:32.956845 10813 leader_election.cc:302] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067; no voters:
I20260812 16:45:32.957333 11313 raft_consensus.cc:2781] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:32.957507 11313 raft_consensus.cc:482] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:32.957624 11313 raft_consensus.cc:3037] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:32.960744 11313 raft_consensus.cc:504] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.961596 11313 leader_election.cc:288] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Requested vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:32.962508 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c69f8209c8454e8fe90e3eab52d930" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d"
I20260812 16:45:32.962594 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c69f8209c8454e8fe90e3eab52d930" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36"
I20260812 16:45:32.962841 11158 raft_consensus.cc:3037] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:32.962942 10691 raft_consensus.cc:3037] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:32.965137 10691 raft_consensus.cc:2454] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:32.965698 10813 leader_election.cc:302] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067; no voters:
I20260812 16:45:32.966041 11313 raft_consensus.cc:2781] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:32.966532 11313 raft_consensus.cc:686] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Becoming Leader. State: Replica: bdec7c95156142049685c77145f9a067, State: Running, Role: LEADER
I20260812 16:45:32.966573 11158 raft_consensus.cc:2454] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:32.966869 11313 consensus_queue.cc:237] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:32.970443 10434 catalog_manager.cc:5495] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdec7c95156142049685c77145f9a067 (127.28.243.2). New cstate: current_term: 1 leader_uuid: "bdec7c95156142049685c77145f9a067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } health_report { overall_health: UNKNOWN } } }
Aug 12 16:45:33 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553129, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.62@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:33.020259 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.005865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:44101 (local address 127.0.0.1:33756)
0812 16:45:33.006447 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:33.006463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:33.006614 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:33.006901 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:33.006905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:33.007633 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:33.007909 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.007919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.008911 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.008916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.009875 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.009888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.010579 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.010585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.010735 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.011523 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:33.011541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:33.013913 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:33.017109 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:33.017126 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:33.017135 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:33.017821 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:33.018354 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:33.018359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:33.018363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:33.018520 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:33.019102 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:33.019110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:33.019288 (+ 178us) client_negotiation.cc:769] Sending connection context
0812 16:45:33.019443 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 16:45:33.019774 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":329,"threads_started":1}
I20260812 16:45:33.020282 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.005781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.28.243.62:44101)
0812 16:45:33.006002 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.006007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.006558 (+ 551us) server_negotiation.cc:407] Connection header received
0812 16:45:33.006725 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.006729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.006784 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.006863 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:33.008054 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.008784 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.010020 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.010402 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.010509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.014114 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:33.014119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:33.014122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:33.014152 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:33.016959 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:33.018066 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:33.018071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:33.018073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:33.018224 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:33.018630 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:33.018634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:33.018636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:33.018957 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:33.019099 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:33.019562 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 16:45:33.019919 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":109}
I20260812 16:45:33.026340 10434 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:33756:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 16:45:33.027908 10434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:33.035481 10665 tablet_service.cc:1467] Processing CreateTablet for tablet 88a64b9df6de44deac6a9d3406b9f4fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=476839e257c44668a2dfefb335d87500]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:45:33.035621 11115 tablet_service.cc:1467] Processing CreateTablet for tablet 88a64b9df6de44deac6a9d3406b9f4fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=476839e257c44668a2dfefb335d87500]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:45:33.036123 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a64b9df6de44deac6a9d3406b9f4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.036154 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a64b9df6de44deac6a9d3406b9f4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.041603 11279 tablet_bootstrap.cc:492] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: Bootstrap starting.
I20260812 16:45:33.043224 11279 tablet_bootstrap.cc:654] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.045684 10888 tablet_service.cc:1467] Processing CreateTablet for tablet 88a64b9df6de44deac6a9d3406b9f4fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=476839e257c44668a2dfefb335d87500]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:45:33.046547 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a64b9df6de44deac6a9d3406b9f4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.047256 11279 tablet_bootstrap.cc:492] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: No bootstrap required, opened a new log
I20260812 16:45:33.047413 11279 ts_tablet_manager.cc:1397] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260812 16:45:33.048027 11279 raft_consensus.cc:348] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.048265 11279 raft_consensus.cc:374] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.048342 11279 raft_consensus.cc:729] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d078536c7641c0b09afd075c329b7d, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.048635 11279 consensus_queue.cc:260] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.049134 11287 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.049414 11287 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:45:33.050716 11279 ts_tablet_manager.cc:1428] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:33.051319 11280 tablet_bootstrap.cc:492] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: Bootstrap starting.
I20260812 16:45:33.053296 11280 tablet_bootstrap.cc:654] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.056830 11282 tablet_bootstrap.cc:492] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: Bootstrap starting.
I20260812 16:45:33.059309 11280 tablet_bootstrap.cc:492] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: No bootstrap required, opened a new log
I20260812 16:45:33.059407 11282 tablet_bootstrap.cc:654] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.060151 11280 ts_tablet_manager.cc:1397] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260812 16:45:33.061213 11280 raft_consensus.cc:348] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.061520 11280 raft_consensus.cc:374] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.061602 11280 raft_consensus.cc:729] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c0533d4f14b9aa1c69bc314d00a36, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.061887 11280 consensus_queue.cc:260] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.062412 11289 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.062716 11289 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:45:33.063408 11280 ts_tablet_manager.cc:1428] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:33.063920 11282 tablet_bootstrap.cc:492] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: No bootstrap required, opened a new log
I20260812 16:45:33.064137 11282 ts_tablet_manager.cc:1397] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 16:45:33.064968 11282 raft_consensus.cc:348] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.065245 11282 raft_consensus.cc:374] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.065338 11282 raft_consensus.cc:729] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdec7c95156142049685c77145f9a067, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.065619 11282 consensus_queue.cc:260] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.066251 11282 ts_tablet_manager.cc:1428] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:45:33.066444 11313 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.066777 11313 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 16:45:33.078064 11233 tablet.cc:2369] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:45:33.078375 11289 raft_consensus.cc:482] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:33.078624 11289 raft_consensus.cc:504] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.079560 11289 leader_election.cc:288] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdec7c95156142049685c77145f9a067 (127.28.243.2:38401), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:33.080273 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a64b9df6de44deac6a9d3406b9f4fc" candidate_uuid: "451c0533d4f14b9aa1c69bc314d00a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdec7c95156142049685c77145f9a067" is_pre_election: true
I20260812 16:45:33.080341 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a64b9df6de44deac6a9d3406b9f4fc" candidate_uuid: "451c0533d4f14b9aa1c69bc314d00a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d" is_pre_election: true
I20260812 16:45:33.080644 10926 raft_consensus.cc:2454] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451c0533d4f14b9aa1c69bc314d00a36 in term 0.
I20260812 16:45:33.080644 11158 raft_consensus.cc:2454] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451c0533d4f14b9aa1c69bc314d00a36 in term 0.
I20260812 16:45:33.081151 10591 leader_election.cc:302] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067; no voters:
I20260812 16:45:33.081436 11289 raft_consensus.cc:2781] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:33.081569 11289 raft_consensus.cc:482] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:33.081669 11289 raft_consensus.cc:3037] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:33.084203 11289 raft_consensus.cc:504] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.084846 11289 leader_election.cc:288] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [CANDIDATE]: Term 1 election: Requested vote from peers bdec7c95156142049685c77145f9a067 (127.28.243.2:38401), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:33.085549 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a64b9df6de44deac6a9d3406b9f4fc" candidate_uuid: "451c0533d4f14b9aa1c69bc314d00a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdec7c95156142049685c77145f9a067"
I20260812 16:45:33.085577 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a64b9df6de44deac6a9d3406b9f4fc" candidate_uuid: "451c0533d4f14b9aa1c69bc314d00a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d"
I20260812 16:45:33.085804 10926 raft_consensus.cc:3037] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:33.085873 11158 raft_consensus.cc:3037] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:33.089298 11158 raft_consensus.cc:2454] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451c0533d4f14b9aa1c69bc314d00a36 in term 1.
I20260812 16:45:33.089336 10926 raft_consensus.cc:2454] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451c0533d4f14b9aa1c69bc314d00a36 in term 1.
I20260812 16:45:33.089943 10591 leader_election.cc:302] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067; no voters:
I20260812 16:45:33.090335 11289 raft_consensus.cc:2781] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:33.090792 11289 raft_consensus.cc:686] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 LEADER]: Becoming Leader. State: Replica: 451c0533d4f14b9aa1c69bc314d00a36, State: Running, Role: LEADER
I20260812 16:45:33.091620 11289 consensus_queue.cc:237] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.094985 11346 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: TxnStatusTablet state changed. Reason: New leader 451c0533d4f14b9aa1c69bc314d00a36. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.095374 11346 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:45:33.096232 11349 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 16:45:33.096575 11349 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 16:45:33.097249 10434 catalog_manager.cc:5495] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1). New cstate: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } health_report { overall_health: UNKNOWN } } }
I20260812 16:45:33.128422 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.121237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.28.243.1:35919)
0812 16:45:33.121433 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.121439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.121611 (+ 172us) server_negotiation.cc:407] Connection header received
0812 16:45:33.121931 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.121935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.122428 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.122594 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:33.123075 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.124129 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.125570 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.126122 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.126271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.126856 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.127671 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.127825 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:33.127979 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 16:45:33.128074 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:45:33.128400 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.121044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35919 (local address 127.0.0.1:60294)
0812 16:45:33.121487 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:33.121507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:33.121706 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:33.122667 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:33.122672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:33.122687 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:45:33.122884 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.122893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.124322 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.124328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.125389 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.125403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.126305 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.126312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.126543 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.126633 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.127803 (+ 1170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.127850 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 16:45:33.127940 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 16:45:33.128118 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260812 16:45:33.133178 11158 raft_consensus.cc:1261] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Refusing update from remote peer 451c0533d4f14b9aa1c69bc314d00a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:33.133339 10926 raft_consensus.cc:1261] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Refusing update from remote peer 451c0533d4f14b9aa1c69bc314d00a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:33.134243 11289 consensus_queue.cc:1035] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:33.134533 11346 consensus_queue.cc:1035] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:33.151934 11287 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: TxnStatusTablet state changed. Reason: New leader 451c0533d4f14b9aa1c69bc314d00a36. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.151942 11313 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: TxnStatusTablet state changed. Reason: New leader 451c0533d4f14b9aa1c69bc314d00a36. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.152302 11287 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:45:33.152323 11313 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:45:33.156047 11306 mvcc.cc:205] Tried to move back new op lower bound from 7317721633305153536 to 7317721633148563456. Current Snapshot: MvccSnapshot[applied={T|T < 7317721633305153536}]
I20260812 16:45:33.159746 11305 mvcc.cc:205] Tried to move back new op lower bound from 7317721633305153536 to 7317721633148563456. Current Snapshot: MvccSnapshot[applied={T|T < 7317721633305153536}]
I20260812 16:45:33.162811 11313 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.163203 11313 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P bdec7c95156142049685c77145f9a067: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:45:33.165019 11289 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.165398 11289 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:45:33.180226 11346 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.180559 11346 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P 451c0533d4f14b9aa1c69bc314d00a36: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:45:33.182317 11287 tablet_replica.cc:439] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "451c0533d4f14b9aa1c69bc314d00a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } } }
I20260812 16:45:33.182601 11287 tablet_replica.cc:442] T 88a64b9df6de44deac6a9d3406b9f4fc P d3d078536c7641c0b09afd075c329b7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:45:33.200495 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.193629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35919 (local address 127.28.243.2:43139)
0812 16:45:33.194342 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:33.194363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:33.194555 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:33.194900 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:33.194905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:33.194945 (+ 40us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:45:33.195238 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.195251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.196371 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.196377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.197592 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.197607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.198986 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.198994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.199230 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.199326 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.199787 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.199800 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 16:45:33.199883 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:45:33.200063 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":418,"threads_started":1}
I20260812 16:45:33.200474 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.193610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:43139 (local address 127.28.243.1:35919)
0812 16:45:33.193807 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.193813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.194459 (+ 646us) server_negotiation.cc:407] Connection header received
0812 16:45:33.194692 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.194698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.194784 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.194861 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:33.195385 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.196210 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.197774 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.198688 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.198840 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.199540 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.199670 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.199749 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:33.199913 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 16:45:33.200005 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:45:33.213572 11158 raft_consensus.cc:1261] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:33.213681 10691 raft_consensus.cc:1261] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:33.214339 11288 consensus_queue.cc:1035] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:33.214829 11288 consensus_queue.cc:1035] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:33.299623 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.277777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44947 (local address 127.28.243.2:38401)
0812 16:45:33.291465 (+ 13688us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.291473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.291894 (+ 421us) server_negotiation.cc:407] Connection header received
0812 16:45:33.292216 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.292223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.292332 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.292410 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:33.293272 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.294553 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.296867 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.297636 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.297750 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.298524 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.298710 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.298789 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:33.299060 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 16:45:33.299169 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13516,"thread_start_us":13303,"threads_started":1}
I20260812 16:45:33.300527 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.277822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:38401 (local address 127.28.243.1:44947)
0812 16:45:33.291779 (+ 13957us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:33.291802 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:33.292034 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:33.292699 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:33.292704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:33.292718 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:45:33.293056 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.293067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.294765 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.294772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.296648 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:33.296671 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.297946 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.297952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:33.298137 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.298223 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.298885 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.298896 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 16:45:33.299974 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0812 16:45:33.300149 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13703,"thread_start_us":14985,"threads_started":1}
I20260812 16:45:33.365948 10433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:33.366251 10433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:33.374521 10433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Sending DeleteTablet for 3 replicas of tablet e8c69f8209c8454e8fe90e3eab52d930
I20260812 16:45:33.376091 10665 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c69f8209c8454e8fe90e3eab52d930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:33 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60284
I20260812 16:45:33.377051 11369 tablet_replica.cc:330] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: stopping tablet replica
I20260812 16:45:33.377049 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c69f8209c8454e8fe90e3eab52d930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:33 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:41264
I20260812 16:45:33.377489 11115 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c69f8209c8454e8fe90e3eab52d930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:33 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:47298
I20260812 16:45:33.377857 11369 raft_consensus.cc:2229] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:33.378222 11369 raft_consensus.cc:2258] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:33.379582 11369 ts_tablet_manager.cc:1905] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:33.388104 11369 ts_tablet_manager.cc:1918] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:45:33.388302 11369 log.cc:1196] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/e8c69f8209c8454e8fe90e3eab52d930
I20260812 16:45:33.388835 11369 ts_tablet_manager.cc:1939] T e8c69f8209c8454e8fe90e3eab52d930 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting consensus metadata
I20260812 16:45:33.390424 10406 catalog_manager.cc:4841] TS 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919): tablet e8c69f8209c8454e8fe90e3eab52d930 (table test-table [id=f0d0effc270f4aca8a315e014bc9d639]) successfully deleted
I20260812 16:45:33.399338 11372 tablet_replica.cc:330] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: stopping tablet replica
I20260812 16:45:33.399595 11372 raft_consensus.cc:2229] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:33.399886 11372 raft_consensus.cc:2258] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:33.400488 11370 tablet_replica.cc:330] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: stopping tablet replica
I20260812 16:45:33.400700 11370 raft_consensus.cc:2229] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:33.401168 11370 raft_consensus.cc:2258] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:33.401221 11372 ts_tablet_manager.cc:1905] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:33.402602 11370 ts_tablet_manager.cc:1905] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:33.407859 11372 ts_tablet_manager.cc:1918] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:45:33.408041 11372 log.cc:1196] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/e8c69f8209c8454e8fe90e3eab52d930
I20260812 16:45:33.408463 11372 ts_tablet_manager.cc:1939] T e8c69f8209c8454e8fe90e3eab52d930 P d3d078536c7641c0b09afd075c329b7d: Deleting consensus metadata
I20260812 16:45:33.409780 10407 catalog_manager.cc:4841] TS d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815): tablet e8c69f8209c8454e8fe90e3eab52d930 (table test-table [id=f0d0effc270f4aca8a315e014bc9d639]) successfully deleted
I20260812 16:45:33.410120 11370 ts_tablet_manager.cc:1918] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:45:33.410281 11370 log.cc:1196] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/e8c69f8209c8454e8fe90e3eab52d930
I20260812 16:45:33.410790 11370 ts_tablet_manager.cc:1939] T e8c69f8209c8454e8fe90e3eab52d930 P bdec7c95156142049685c77145f9a067: Deleting consensus metadata
I20260812 16:45:33.412139 10407 catalog_manager.cc:4841] TS bdec7c95156142049685c77145f9a067 (127.28.243.2:38401): tablet e8c69f8209c8454e8fe90e3eab52d930 (table test-table [id=f0d0effc270f4aca8a315e014bc9d639]) successfully deleted
Aug 12 16:45:33 dist-test-slave-1vmg krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:45:33 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:33 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 16:45:33.470062 11044 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:33.455587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.28.243.62:44101)
0812 16:45:33.455824 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.455830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.456345 (+ 515us) server_negotiation.cc:407] Connection header received
0812 16:45:33.456478 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.456483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.456588 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.456733 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:33.458771 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.459686 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.461728 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.462459 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.462601 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.469777 (+ 7176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33768: BlockingRecv error: recv got EOF from 127.0.0.1:33768 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 16:45:33 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM
I20260812 16:45:33.494131 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.477176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.28.243.62:44101)
0812 16:45:33.477363 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.477367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.477824 (+ 457us) server_negotiation.cc:407] Connection header received
0812 16:45:33.478008 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.478014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.478099 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.478191 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:33.479751 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.480564 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.481962 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.482426 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.482547 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.487388 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:33.487395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:33.487400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:33.487452 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:33.490654 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:33.491843 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:33.491849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:33.491851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:33.492041 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:33.492550 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:33.492554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:33.492557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:33.492951 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:33.493164 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:33.493591 (+ 427us) server_negotiation.cc:299] Negotiation successful
0812 16:45:33.493795 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:45:33.499796 10433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:33.500530 10433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:33.509430 10888 tablet_service.cc:1467] Processing CreateTablet for tablet 0e097a8b8a2b4acdbabf1a243d1030f7 (DEFAULT_TABLE table=test-table [id=5516c4f068d54cb1afa793b94798cc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:33.509430 11115 tablet_service.cc:1467] Processing CreateTablet for tablet 0e097a8b8a2b4acdbabf1a243d1030f7 (DEFAULT_TABLE table=test-table [id=5516c4f068d54cb1afa793b94798cc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:33.509553 10665 tablet_service.cc:1467] Processing CreateTablet for tablet 0e097a8b8a2b4acdbabf1a243d1030f7 (DEFAULT_TABLE table=test-table [id=5516c4f068d54cb1afa793b94798cc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:33.509970 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e097a8b8a2b4acdbabf1a243d1030f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.509991 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e097a8b8a2b4acdbabf1a243d1030f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.510077 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e097a8b8a2b4acdbabf1a243d1030f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.517405 11279 tablet_bootstrap.cc:492] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Bootstrap starting.
I20260812 16:45:33.517405 11282 tablet_bootstrap.cc:492] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Bootstrap starting.
I20260812 16:45:33.517441 11280 tablet_bootstrap.cc:492] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Bootstrap starting.
I20260812 16:45:33.519382 11280 tablet_bootstrap.cc:654] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.519464 11279 tablet_bootstrap.cc:654] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.519538 11282 tablet_bootstrap.cc:654] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.531919 11279 tablet_bootstrap.cc:492] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: No bootstrap required, opened a new log
I20260812 16:45:33.532116 11279 ts_tablet_manager.cc:1397] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260812 16:45:33.533319 11279 raft_consensus.cc:348] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.533608 11279 raft_consensus.cc:374] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.533699 11279 raft_consensus.cc:729] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d078536c7641c0b09afd075c329b7d, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.533986 11279 consensus_queue.cc:260] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.534564 11282 tablet_bootstrap.cc:492] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: No bootstrap required, opened a new log
I20260812 16:45:33.534786 11282 ts_tablet_manager.cc:1397] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260812 16:45:33.537020 11282 raft_consensus.cc:348] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.537338 11279 ts_tablet_manager.cc:1428] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260812 16:45:33.537379 11282 raft_consensus.cc:374] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.537586 11282 raft_consensus.cc:729] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdec7c95156142049685c77145f9a067, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.537942 11282 consensus_queue.cc:260] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.541585 11282 ts_tablet_manager.cc:1428] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 16:45:33.542712 11280 tablet_bootstrap.cc:492] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: No bootstrap required, opened a new log
I20260812 16:45:33.542909 11280 ts_tablet_manager.cc:1397] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20260812 16:45:33.543952 11280 raft_consensus.cc:348] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.544302 11280 raft_consensus.cc:374] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.544436 11280 raft_consensus.cc:729] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c0533d4f14b9aa1c69bc314d00a36, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.544765 11280 consensus_queue.cc:260] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.545493 11280 ts_tablet_manager.cc:1428] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:45:33.644763 11301 raft_consensus.cc:482] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:33.645021 11301 raft_consensus.cc:504] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.645726 11301 leader_election.cc:288] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:33.646453 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e097a8b8a2b4acdbabf1a243d1030f7" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d" is_pre_election: true
I20260812 16:45:33.646466 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e097a8b8a2b4acdbabf1a243d1030f7" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36" is_pre_election: true
I20260812 16:45:33.646741 11158 raft_consensus.cc:2454] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:33.646742 10691 raft_consensus.cc:2454] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:33.647542 10813 leader_election.cc:302] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067; no voters:
I20260812 16:45:33.648093 11301 raft_consensus.cc:2781] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:33.648223 11301 raft_consensus.cc:482] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:33.648335 11301 raft_consensus.cc:3037] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:33.650717 11301 raft_consensus.cc:504] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.651434 11301 leader_election.cc:288] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Requested vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:33.652454 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e097a8b8a2b4acdbabf1a243d1030f7" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36"
I20260812 16:45:33.652426 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e097a8b8a2b4acdbabf1a243d1030f7" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d"
I20260812 16:45:33.652704 10691 raft_consensus.cc:3037] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:33.652714 11158 raft_consensus.cc:3037] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:33.654855 10691 raft_consensus.cc:2454] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:33.655455 10813 leader_election.cc:302] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067; no voters:
I20260812 16:45:33.655701 11158 raft_consensus.cc:2454] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:33.655792 11301 raft_consensus.cc:2781] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:33.655977 11301 raft_consensus.cc:686] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Becoming Leader. State: Replica: bdec7c95156142049685c77145f9a067, State: Running, Role: LEADER
I20260812 16:45:33.656344 11301 consensus_queue.cc:237] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.659406 10434 catalog_manager.cc:5495] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdec7c95156142049685c77145f9a067 (127.28.243.2). New cstate: current_term: 1 leader_uuid: "bdec7c95156142049685c77145f9a067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } health_report { overall_health: UNKNOWN } } }
I20260812 16:45:33.702463 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:33.695320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.28.243.2:38401)
0812 16:45:33.695522 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:33.695527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:33.695894 (+ 367us) server_negotiation.cc:407] Connection header received
0812 16:45:33.696106 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:33.696110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:33.696193 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:33.696295 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:45:33.696828 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.697644 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.699890 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:33.700516 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:33.700646 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:33.701658 (+ 1012us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:45:33.701808 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:45:33.701901 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:33.702072 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 16:45:33.702170 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:45:33.706526 11158 raft_consensus.cc:1261] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:33.706526 10691 raft_consensus.cc:1261] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:33.707262 11301 consensus_queue.cc:1035] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:33.707624 11288 consensus_queue.cc:1035] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:33.725387 10433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:33.725759 10433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:33.730077 10433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Sending DeleteTablet for 3 replicas of tablet 0e097a8b8a2b4acdbabf1a243d1030f7
I20260812 16:45:33.731356 10665 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e097a8b8a2b4acdbabf1a243d1030f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:33 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60284
I20260812 16:45:33.731585 11115 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e097a8b8a2b4acdbabf1a243d1030f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:33 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:47298
I20260812 16:45:33.731762 11369 tablet_replica.cc:330] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: stopping tablet replica
I20260812 16:45:33.731906 11372 tablet_replica.cc:330] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: stopping tablet replica
I20260812 16:45:33.732002 11369 raft_consensus.cc:2229] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:33.732124 11372 raft_consensus.cc:2229] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:33.732290 11369 raft_consensus.cc:2258] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:33.732388 11372 raft_consensus.cc:2258] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:33.732503 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e097a8b8a2b4acdbabf1a243d1030f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:33 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:41264
I20260812 16:45:33.732976 11370 tablet_replica.cc:330] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: stopping tablet replica
I20260812 16:45:33.733304 11370 raft_consensus.cc:2229] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:33.733711 11369 ts_tablet_manager.cc:1905] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:33.733793 11372 ts_tablet_manager.cc:1905] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:33.733806 11370 raft_consensus.cc:2258] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:33.734441 10433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:45:33.735313 10433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:45:33.737727 11370 ts_tablet_manager.cc:1905] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:33.741156 11369 ts_tablet_manager.cc:1918] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:33.741376 11369 log.cc:1196] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/0e097a8b8a2b4acdbabf1a243d1030f7
I20260812 16:45:33.741819 11369 ts_tablet_manager.cc:1939] T 0e097a8b8a2b4acdbabf1a243d1030f7 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting consensus metadata
I20260812 16:45:33.743410 11372 ts_tablet_manager.cc:1918] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:33.743618 11372 log.cc:1196] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/0e097a8b8a2b4acdbabf1a243d1030f7
I20260812 16:45:33.744040 11372 ts_tablet_manager.cc:1939] T 0e097a8b8a2b4acdbabf1a243d1030f7 P d3d078536c7641c0b09afd075c329b7d: Deleting consensus metadata
I20260812 16:45:33.744535 10406 catalog_manager.cc:4841] TS 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919): tablet 0e097a8b8a2b4acdbabf1a243d1030f7 (table test-table [id=5516c4f068d54cb1afa793b94798cc44]) successfully deleted
I20260812 16:45:33.745766 10407 catalog_manager.cc:4841] TS d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815): tablet 0e097a8b8a2b4acdbabf1a243d1030f7 (table test-table [id=5516c4f068d54cb1afa793b94798cc44]) successfully deleted
I20260812 16:45:33.746434 11370 ts_tablet_manager.cc:1918] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:45:33.746640 11370 log.cc:1196] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/0e097a8b8a2b4acdbabf1a243d1030f7
I20260812 16:45:33.747120 11370 ts_tablet_manager.cc:1939] T 0e097a8b8a2b4acdbabf1a243d1030f7 P bdec7c95156142049685c77145f9a067: Deleting consensus metadata
I20260812 16:45:33.747725 10665 tablet_service.cc:1467] Processing CreateTablet for tablet 369eed53e16449f59fc30bdcd24fd403 (DEFAULT_TABLE table=test-table [id=51c5804d3f6641ce911e3e3470a8e6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:33.748217 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369eed53e16449f59fc30bdcd24fd403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.748219 10888 tablet_service.cc:1467] Processing CreateTablet for tablet 369eed53e16449f59fc30bdcd24fd403 (DEFAULT_TABLE table=test-table [id=51c5804d3f6641ce911e3e3470a8e6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:33.748591 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369eed53e16449f59fc30bdcd24fd403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.748699 11115 tablet_service.cc:1467] Processing CreateTablet for tablet 369eed53e16449f59fc30bdcd24fd403 (DEFAULT_TABLE table=test-table [id=51c5804d3f6641ce911e3e3470a8e6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:45:33.749490 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369eed53e16449f59fc30bdcd24fd403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:33.753538 11282 tablet_bootstrap.cc:492] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Bootstrap starting.
I20260812 16:45:33.754804 10407 catalog_manager.cc:4841] TS bdec7c95156142049685c77145f9a067 (127.28.243.2:38401): tablet 0e097a8b8a2b4acdbabf1a243d1030f7 (table test-table [id=5516c4f068d54cb1afa793b94798cc44]) successfully deleted
I20260812 16:45:33.755569 11282 tablet_bootstrap.cc:654] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.757622 11282 tablet_bootstrap.cc:492] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: No bootstrap required, opened a new log
I20260812 16:45:33.758184 11282 ts_tablet_manager.cc:1397] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 16:45:33.759291 11280 tablet_bootstrap.cc:492] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Bootstrap starting.
I20260812 16:45:33.759552 11279 tablet_bootstrap.cc:492] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Bootstrap starting.
I20260812 16:45:33.759512 11282 raft_consensus.cc:348] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.760264 11282 raft_consensus.cc:374] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.760468 11282 raft_consensus.cc:729] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdec7c95156142049685c77145f9a067, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.760842 11282 consensus_queue.cc:260] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.761142 11280 tablet_bootstrap.cc:654] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.761700 11279 tablet_bootstrap.cc:654] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:33.763350 11282 ts_tablet_manager.cc:1428] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 16:45:33.763597 11280 tablet_bootstrap.cc:492] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: No bootstrap required, opened a new log
I20260812 16:45:33.763787 11280 ts_tablet_manager.cc:1397] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260812 16:45:33.764611 11280 raft_consensus.cc:348] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.764884 11280 raft_consensus.cc:374] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.764973 11280 raft_consensus.cc:729] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c0533d4f14b9aa1c69bc314d00a36, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.765108 11279 tablet_bootstrap.cc:492] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: No bootstrap required, opened a new log
I20260812 16:45:33.765312 11279 ts_tablet_manager.cc:1397] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260812 16:45:33.765272 11280 consensus_queue.cc:260] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.765769 11280 ts_tablet_manager.cc:1428] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:45:33.766764 11279 raft_consensus.cc:348] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.767271 11279 raft_consensus.cc:374] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:33.767396 11279 raft_consensus.cc:729] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d078536c7641c0b09afd075c329b7d, State: Initialized, Role: FOLLOWER
I20260812 16:45:33.767688 11279 consensus_queue.cc:260] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:33.768303 11279 ts_tablet_manager.cc:1428] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:34.024709 11288 raft_consensus.cc:482] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:45:34.024983 11288 raft_consensus.cc:504] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:34.025638 11288 leader_election.cc:288] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:34.026391 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369eed53e16449f59fc30bdcd24fd403" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36" is_pre_election: true
I20260812 16:45:34.026408 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369eed53e16449f59fc30bdcd24fd403" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d" is_pre_election: true
I20260812 16:45:34.026674 11158 raft_consensus.cc:2454] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:34.026676 10691 raft_consensus.cc:2454] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 0.
I20260812 16:45:34.027274 10817 leader_election.cc:302] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 451c0533d4f14b9aa1c69bc314d00a36, bdec7c95156142049685c77145f9a067, d3d078536c7641c0b09afd075c329b7d; no voters:
I20260812 16:45:34.027626 11288 raft_consensus.cc:2781] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:45:34.027726 11288 raft_consensus.cc:482] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:45:34.027830 11288 raft_consensus.cc:3037] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:34.030128 11288 raft_consensus.cc:504] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:34.030843 11288 leader_election.cc:288] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Requested vote from peers 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919), d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815)
I20260812 16:45:34.031617 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369eed53e16449f59fc30bdcd24fd403" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d078536c7641c0b09afd075c329b7d"
I20260812 16:45:34.031652 10691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369eed53e16449f59fc30bdcd24fd403" candidate_uuid: "bdec7c95156142049685c77145f9a067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c0533d4f14b9aa1c69bc314d00a36"
I20260812 16:45:34.031849 11158 raft_consensus.cc:3037] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:34.031879 10691 raft_consensus.cc:3037] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:34.034296 11158 raft_consensus.cc:2454] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:34.035032 10817 leader_election.cc:302] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdec7c95156142049685c77145f9a067, d3d078536c7641c0b09afd075c329b7d; no voters:
I20260812 16:45:34.035495 11288 raft_consensus.cc:2781] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:34.035723 11288 raft_consensus.cc:686] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Becoming Leader. State: Replica: bdec7c95156142049685c77145f9a067, State: Running, Role: LEADER
I20260812 16:45:34.035961 10691 raft_consensus.cc:2454] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdec7c95156142049685c77145f9a067 in term 1.
I20260812 16:45:34.036026 11288 consensus_queue.cc:237] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } }
I20260812 16:45:34.039795 10434 catalog_manager.cc:5495] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdec7c95156142049685c77145f9a067 (127.28.243.2). New cstate: current_term: 1 leader_uuid: "bdec7c95156142049685c77145f9a067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdec7c95156142049685c77145f9a067" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 38401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 } health_report { overall_health: UNKNOWN } } }
I20260812 16:45:34.088145 11158 raft_consensus.cc:1261] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:34.088181 10691 raft_consensus.cc:1261] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Refusing update from remote peer bdec7c95156142049685c77145f9a067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:45:34.089141 11301 consensus_queue.cc:1035] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d078536c7641c0b09afd075c329b7d" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:34.089767 11301 consensus_queue.cc:1035] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451c0533d4f14b9aa1c69bc314d00a36" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:45:34.150470 10434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:34.150722 10434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:45:34.153957 10434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c8c75fb558491a8313ab69e884574c: Sending DeleteTablet for 3 replicas of tablet 369eed53e16449f59fc30bdcd24fd403
I20260812 16:45:34.154896 10665 tablet_service.cc:1514] Processing DeleteTablet for tablet 369eed53e16449f59fc30bdcd24fd403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:34 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60284
I20260812 16:45:34.154945 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet 369eed53e16449f59fc30bdcd24fd403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:34 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:41264
I20260812 16:45:34.155042 11115 tablet_service.cc:1514] Processing DeleteTablet for tablet 369eed53e16449f59fc30bdcd24fd403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:45:34 UTC) from {username='oryx', principal='oryx/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:47298
I20260812 16:45:34.155275 11370 tablet_replica.cc:330] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: stopping tablet replica
I20260812 16:45:34.155308 11369 tablet_replica.cc:330] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: stopping tablet replica
I20260812 16:45:34.155319 11372 tablet_replica.cc:330] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: stopping tablet replica
I20260812 16:45:34.155459 11370 raft_consensus.cc:2229] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:45:34.155490 11372 raft_consensus.cc:2229] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:34.155489 11369 raft_consensus.cc:2229] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:45:34.155696 11369 raft_consensus.cc:2258] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:34.155696 11372 raft_consensus.cc:2258] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:34.155766 11370 raft_consensus.cc:2258] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:45:34.156868 11370 ts_tablet_manager.cc:1905] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:34.157052 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 10566
I20260812 16:45:34.157143 11372 ts_tablet_manager.cc:1905] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:34.158862 11369 ts_tablet_manager.cc:1905] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:45:34.163702 11370 ts_tablet_manager.cc:1918] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:45:34.163851 11370 log.cc:1196] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/369eed53e16449f59fc30bdcd24fd403
I20260812 16:45:34.164125 11372 ts_tablet_manager.cc:1918] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:45:34.164268 11370 ts_tablet_manager.cc:1939] T 369eed53e16449f59fc30bdcd24fd403 P bdec7c95156142049685c77145f9a067: Deleting consensus metadata
I20260812 16:45:34.164278 11372 log.cc:1196] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/369eed53e16449f59fc30bdcd24fd403
I20260812 16:45:34.164659 11372 ts_tablet_manager.cc:1939] T 369eed53e16449f59fc30bdcd24fd403 P d3d078536c7641c0b09afd075c329b7d: Deleting consensus metadata
I20260812 16:45:34.165613 10407 catalog_manager.cc:4841] TS bdec7c95156142049685c77145f9a067 (127.28.243.2:38401): tablet 369eed53e16449f59fc30bdcd24fd403 (table test-table [id=51c5804d3f6641ce911e3e3470a8e6d4]) successfully deleted
I20260812 16:45:34.165946 10407 catalog_manager.cc:4841] TS d3d078536c7641c0b09afd075c329b7d (127.28.243.3:34815): tablet 369eed53e16449f59fc30bdcd24fd403 (table test-table [id=51c5804d3f6641ce911e3e3470a8e6d4]) successfully deleted
I20260812 16:45:34.168560 11369 ts_tablet_manager.cc:1918] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:45:34.168706 11369 log.cc:1196] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/369eed53e16449f59fc30bdcd24fd403
I20260812 16:45:34.169106 11369 ts_tablet_manager.cc:1939] T 369eed53e16449f59fc30bdcd24fd403 P 451c0533d4f14b9aa1c69bc314d00a36: Deleting consensus metadata
I20260812 16:45:34.170219 10406 catalog_manager.cc:4841] TS 451c0533d4f14b9aa1c69bc314d00a36 (127.28.243.1:35919): tablet 369eed53e16449f59fc30bdcd24fd403 (table test-table [id=51c5804d3f6641ce911e3e3470a8e6d4]) successfully deleted
Aug 12 16:45:34 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.1@KRBTEST.COM
I20260812 16:45:34.188549 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:34.158303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.28.243.1:35919)
0812 16:45:34.166221 (+ 7918us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:34.166228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:34.166733 (+ 505us) server_negotiation.cc:407] Connection header received
0812 16:45:34.166964 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:34.166970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:34.167069 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:34.167185 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:34.169544 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:34.170368 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:34.171942 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:34.172496 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:34.172681 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:34.181317 (+ 8636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:34.181345 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:34.181370 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:34.181425 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:34.184952 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:34.186234 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:34.186267 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:34.186271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:34.186485 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:34.186845 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:34.186848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:34.186850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:34.187362 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:34.187625 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:34.187925 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 16:45:34.188151 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7742,"thread_start_us":7575,"threads_started":1}
I20260812 16:45:34.189663 10772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:34.329847 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 10566
I20260812 16:45:34.372849 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 10786
Aug 12 16:45:34 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.2@KRBTEST.COM
I20260812 16:45:34.390642 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:34.373783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.28.243.2:38401)
0812 16:45:34.374303 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:34.374310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:34.374381 (+ 71us) server_negotiation.cc:407] Connection header received
0812 16:45:34.374590 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:34.374596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:34.374684 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:34.374799 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:34.376699 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:34.377641 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:34.379399 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:34.379917 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:34.380055 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:34.384969 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:34.384987 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:34.385005 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:34.385051 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:34.388072 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:34.388769 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:34.388776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:34.388779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:34.388911 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:34.389253 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:34.389256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:34.389258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:34.389656 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:34.389895 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:34.390168 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 16:45:34.390364 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260812 16:45:34.391350 10993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:34.526122 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 10786
I20260812 16:45:34.561604 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 11007
Aug 12 16:45:34 dist-test-slave-1vmg krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.3@KRBTEST.COM
I20260812 16:45:34.582129 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:34.562490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.28.243.3:34815)
0812 16:45:34.563352 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:34.563356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:34.563374 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:45:34.563430 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:34.563434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:34.563499 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:34.563608 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:34.565187 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:34.566164 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:34.568255 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:34.568763 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:34.568895 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:34.575172 (+ 6277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:34.575194 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:34.575214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:34.575270 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:34.578412 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:34.579587 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:34.579596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:34.579599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:34.579802 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:34.580342 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:34.580347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:34.580349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:34.580860 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:34.581155 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:34.581561 (+ 406us) server_negotiation.cc:299] Negotiation successful
0812 16:45:34.581768 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":592,"threads_started":1}
I20260812 16:45:34.583148 11219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:34.689302 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 11007
I20260812 16:45:34.720465 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 10385
I20260812 16:45:34.721912 10501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:34.848594 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 10385
2026-08-12T16:45:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7820 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:45:35 dist-test-slave-1vmg krb5kdc[11436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:45:35 dist-test-slave-1vmg krb5kdc[11436](info): set up 2 sockets
Aug 12 16:45:35 dist-test-slave-1vmg krb5kdc[11436](info): commencing operation
krb5kdc: starting...
W20260812 16:45:36.205520 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:45:36 dist-test-slave-1vmg krb5kdc[11436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:45:36.545169 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:41781
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:36.940255 11456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:36.940604 11456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:36.940734 11456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:36.948984 11456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:36.949172 11456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:36.949257 11456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:36.949335 11456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:36.949402 11456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:36.960682 11456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:41781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:36.962258 11456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:36.963727 11456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:36.977407 11465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:36.977447 11463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:36.977783 11462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:36.978180 11456 server_base.cc:1034] running on GCE node
I20260812 16:45:36.979147 11456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:36.981207 11456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:36.984776 11456 hybrid_clock.cc:648] HybridClock initialized: now 1786553136984654 us; error 87 us; skew 500 ppm
I20260812 16:45:36.985487 11456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T16:45:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2195 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:45:37 dist-test-slave-1vmg krb5kdc[11472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:45:37 dist-test-slave-1vmg krb5kdc[11472](info): set up 2 sockets
Aug 12 16:45:37 dist-test-slave-1vmg krb5kdc[11472](info): commencing operation
krb5kdc: starting...
W20260812 16:45:38.405503 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:45:38 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:45:38.743811 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42539
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:39.180374 11496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:39.180704 11496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:39.180796 11496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:39.189391 11496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:39.189607 11496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:39.189664 11496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:39.189725 11496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:39.189787 11496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:39.203148 11496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:42539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:39.204663 11496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:39.206139 11496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:39.222402 11506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:39.222368 11507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:39.222302 11512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:39.223265 11496 server_base.cc:1034] running on GCE node
I20260812 16:45:39.224206 11496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:39.225901 11496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:39.227241 11496 hybrid_clock.cc:648] HybridClock initialized: now 1786553139227141 us; error 45 us; skew 500 ppm
I20260812 16:45:39.227895 11496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:39 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:39.239105 11496 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:45:39.241130 11496 webserver.cc:460] Webserver started at http://127.28.243.62:35453/ using document root <none> and password file <none>
I20260812 16:45:39.241992 11496 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:39.242147 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:39.242516 11496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:39.246088 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "4913db1e283e444ea8f19d40b279c060"
format_stamp: "Formatted at 2026-08-12 16:45:39 on dist-test-slave-1vmg"
server_key: "769610e056a7241990fb27c4d4ab6212"
server_key_iv: "8874bb1579efee4aa8348eb77b5d29f7"
server_key_version: "encryptionkey@0"
I20260812 16:45:39.247123 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "4913db1e283e444ea8f19d40b279c060"
format_stamp: "Formatted at 2026-08-12 16:45:39 on dist-test-slave-1vmg"
server_key: "769610e056a7241990fb27c4d4ab6212"
server_key_iv: "8874bb1579efee4aa8348eb77b5d29f7"
server_key_version: "encryptionkey@0"
I20260812 16:45:39.255468 11496 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260812 16:45:39.261303 11520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:39.263267 11496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260812 16:45:39.263521 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "4913db1e283e444ea8f19d40b279c060"
format_stamp: "Formatted at 2026-08-12 16:45:39 on dist-test-slave-1vmg"
server_key: "769610e056a7241990fb27c4d4ab6212"
server_key_iv: "8874bb1579efee4aa8348eb77b5d29f7"
server_key_version: "encryptionkey@0"
I20260812 16:45:39.263784 11496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:39.289805 11496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:39.297016 11496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:39.297487 11496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:39.320472 11496 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:42539
I20260812 16:45:39.320482 11599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:42539 every 8 connection(s)
I20260812 16:45:39.322037 11496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:39.326786 11600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:45:39.332158 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 11496
I20260812 16:45:39.332450 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:45:39.333458 29644 external_mini_cluster.cc:1428] Setting key 5cbc3aca7c8d0e33bad10deefe814838
I20260812 16:45:39.345770 11600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Bootstrap starting.
I20260812 16:45:39.350950 11600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Neither blocks nor log segments found. Creating new log.
I20260812 16:45:39.352720 11600 log.cc:824] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:45:39 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:39.357280 11600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: No bootstrap required, opened a new log
I20260812 16:45:39.367749 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:39.337060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.28.243.62:42539)
0812 16:45:39.338195 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:39.338211 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:39.338270 (+ 59us) server_negotiation.cc:407] Connection header received
0812 16:45:39.340501 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:39.340541 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:39.341234 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:39.341762 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:39.344153 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:39.345792 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:39.348078 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:39.349190 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:39.349597 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:39.356667 (+ 7070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:39.356682 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:39.356707 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:39.356774 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:39.361397 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:39.362491 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:39.362502 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:39.362512 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:39.362777 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:39.363304 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:39.363309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:39.363312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:39.364174 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:39.364529 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:39.365040 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 16:45:39.365606 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":515,"threads_started":1}
I20260812 16:45:39.369343 11600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:39.370035 11600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:45:39.370158 11600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4913db1e283e444ea8f19d40b279c060, State: Initialized, Role: FOLLOWER
I20260812 16:45:39.370980 11600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:39.371294 11600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:39.371766 11600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:39.372036 11600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:45:39.375139 11600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:39.376052 11600 leader_election.cc:302] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4913db1e283e444ea8f19d40b279c060; no voters:
I20260812 16:45:39.377044 11600 leader_election.cc:288] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:45:39.377198 11606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:45:39.378427 11606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 LEADER]: Becoming Leader. State: Replica: 4913db1e283e444ea8f19d40b279c060, State: Running, Role: LEADER
I20260812 16:45:39.379171 11606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:39.379694 11600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:39.382738 11607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4913db1e283e444ea8f19d40b279c060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } } }
I20260812 16:45:39.383294 11607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:39.387696 11610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4913db1e283e444ea8f19d40b279c060. Latest consensus state: current_term: 1 leader_uuid: "4913db1e283e444ea8f19d40b279c060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } } }
I20260812 16:45:39.388034 11610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:39.392500 11621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:39.399724 11621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:39.411182 11621 catalog_manager.cc:1347] Generated new cluster ID: d1bfbc2e58a74499875ce31dee356aeb
I20260812 16:45:39.411490 11621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:39.426617 11621 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:45:39.430109 11621 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:39.442668 11621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Generated new TSK 0
I20260812 16:45:39.443996 11621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:45:39.561213 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42539
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:39.974476 11642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:39.974784 11642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:39.974905 11642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:39.983144 11642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:39.983304 11642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:39.983470 11642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:39.995245 11642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:39.996703 11642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:39.998140 11642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:40.013458 11647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:40.013415 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:40.013356 11651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:40.014585 11642 server_base.cc:1034] running on GCE node
I20260812 16:45:40.015491 11642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:40.016737 11642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:40.018219 11642 hybrid_clock.cc:648] HybridClock initialized: now 1786553140018154 us; error 54 us; skew 500 ppm
I20260812 16:45:40.019397 11642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:40 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:40.022984 11642 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:45:40.024655 11642 webserver.cc:460] Webserver started at http://127.28.243.1:46771/ using document root <none> and password file <none>
I20260812 16:45:40.025427 11642 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:40.025566 11642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:40.025882 11642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:40.029345 11642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "edfb7a4465af4ac18d12452ca8d1f998"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "d2f337d35c788446f04ffea0044c4f05"
server_key_iv: "3338620e637a5f5f0b18381d23f44c45"
server_key_version: "encryptionkey@0"
I20260812 16:45:40.030331 11642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "edfb7a4465af4ac18d12452ca8d1f998"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "d2f337d35c788446f04ffea0044c4f05"
server_key_iv: "3338620e637a5f5f0b18381d23f44c45"
server_key_version: "encryptionkey@0"
I20260812 16:45:40.036582 11642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260812 16:45:40.042024 11667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:40.043992 11642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:45:40.044214 11642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "edfb7a4465af4ac18d12452ca8d1f998"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "d2f337d35c788446f04ffea0044c4f05"
server_key_iv: "3338620e637a5f5f0b18381d23f44c45"
server_key_version: "encryptionkey@0"
I20260812 16:45:40.044451 11642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:40.065806 11642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:40.071544 11642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:40.071899 11642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:40.072983 11642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:40.074546 11642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:40.074645 11642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:40.074810 11642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:40.074868 11642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:40.111737 11642 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:32881
I20260812 16:45:40.111778 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:32881 every 8 connection(s)
I20260812 16:45:40.113144 11642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:40.119429 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 11642
I20260812 16:45:40.119623 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:45:40.120298 29644 external_mini_cluster.cc:1428] Setting key f8d91df97652ae6cda65d48a2e66652f
Aug 12 16:45:40 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:40.137399 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:40.116042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:34683 (local address 127.28.243.62:42539)
0812 16:45:40.116559 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:40.116564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:40.117955 (+ 1391us) server_negotiation.cc:407] Connection header received
0812 16:45:40.120142 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:40.120148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:40.120240 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:40.120342 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:40.122461 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.123580 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.125502 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.126072 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.126225 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:40.130239 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:40.130246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:40.130251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:40.130298 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:40.133639 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:40.134656 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:40.134661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:40.134664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:40.134807 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:40.135410 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:40.135414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:40.135417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:40.135745 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:40.135878 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:40.136866 (+ 988us) server_negotiation.cc:299] Negotiation successful
0812 16:45:40.137101 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
I20260812 16:45:40.138862 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:40.116657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:34683)
0812 16:45:40.117762 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:40.117807 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:40.119798 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:40.120630 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:40.120639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:40.121345 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:40.122254 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:40.122286 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.123810 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.123817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:40.125308 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:40.125326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.126376 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.126382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:40.126620 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:40.127470 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:40.127503 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:40.129926 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:40.133897 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:40.133907 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:40.133924 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:40.134482 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:40.134947 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:40.134952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:40.134955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:40.135258 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:40.135887 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:40.135897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:40.136414 (+ 517us) client_negotiation.cc:769] Sending connection context
0812 16:45:40.136748 (+ 334us) client_negotiation.cc:240] Negotiation successful
0812 16:45:40.137368 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":434,"threads_started":1}
I20260812 16:45:40.141673 11844 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42539
I20260812 16:45:40.142366 11844 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:40.143496 11844 heartbeater.cc:511] Master 127.28.243.62:42539 requested a full tablet report, sending...
I20260812 16:45:40.147195 11546 ts_manager.cc:194] Registered new tserver with Master: edfb7a4465af4ac18d12452ca8d1f998 (127.28.243.1:32881)
I20260812 16:45:40.149278 11546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:34683
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:45:40.191356 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42539
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:40.566608 11853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:40.566967 11853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:40.567121 11853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:40.574709 11853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:40.574857 11853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:40.575044 11853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:40.586217 11853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:40.588542 11853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:40.590054 11853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:40.605170 11862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:40.605383 11865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:40.605399 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:40.607125 11853 server_base.cc:1034] running on GCE node
I20260812 16:45:40.608348 11853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:40.609639 11853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:40.611021 11853 hybrid_clock.cc:648] HybridClock initialized: now 1786553140610861 us; error 51 us; skew 500 ppm
I20260812 16:45:40.611439 11853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:40 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:40.615171 11853 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:45:40.616926 11853 webserver.cc:460] Webserver started at http://127.28.243.2:40881/ using document root <none> and password file <none>
I20260812 16:45:40.617755 11853 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:40.617882 11853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:40.618227 11853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:40.621559 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "b83e3dde9d9e4d348afadbe2e7a05dbe"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "bd1389303fcc83e0888b9209314e5c79"
server_key_iv: "ade8c14cdc452b5285e1d6f01c10b52d"
server_key_version: "encryptionkey@0"
I20260812 16:45:40.623270 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "b83e3dde9d9e4d348afadbe2e7a05dbe"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "bd1389303fcc83e0888b9209314e5c79"
server_key_iv: "ade8c14cdc452b5285e1d6f01c10b52d"
server_key_version: "encryptionkey@0"
I20260812 16:45:40.629832 11853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:45:40.634133 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:40.635864 11853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:45:40.636059 11853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "b83e3dde9d9e4d348afadbe2e7a05dbe"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "bd1389303fcc83e0888b9209314e5c79"
server_key_iv: "ade8c14cdc452b5285e1d6f01c10b52d"
server_key_version: "encryptionkey@0"
I20260812 16:45:40.636300 11853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:40.671674 11853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:40.676506 11853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:40.676816 11853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:40.678031 11853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:40.679651 11853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:40.679745 11853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:40.679877 11853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:40.679944 11853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:40.718150 11853 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:43111
I20260812 16:45:40.718281 12063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:43111 every 8 connection(s)
I20260812 16:45:40.719712 11853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:40.727926 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 11853
I20260812 16:45:40.728140 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:45:40.728688 29644 external_mini_cluster.cc:1428] Setting key 9739a31a15e6a9caa2a1b8231b647653
Aug 12 16:45:40 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:40.743893 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:40.722639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:44581 (local address 127.28.243.62:42539)
0812 16:45:40.723110 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:40.723115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:40.724176 (+ 1061us) server_negotiation.cc:407] Connection header received
0812 16:45:40.726195 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:40.726201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:40.726295 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:40.726397 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:40.728590 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.729393 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.731329 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.731960 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.732090 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:40.736020 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:40.736026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:40.736031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:40.736076 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:40.739347 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:40.740303 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:40.740309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:40.740312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:40.740502 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:40.741569 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:40.741575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:40.741578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:40.742001 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:40.742185 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:40.743172 (+ 987us) server_negotiation.cc:299] Negotiation successful
0812 16:45:40.743462 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260812 16:45:40.745308 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:40.723159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:44581)
0812 16:45:40.723957 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:40.724003 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:40.725854 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:40.726697 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:40.726707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:40.727385 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:40.728323 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:40.728356 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.729606 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.729612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:40.731164 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:40.731181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:40.732219 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:40.732224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:40.732447 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:40.733180 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:40.733207 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:40.735708 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:40.739548 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:40.739558 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:40.739575 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:40.740154 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:40.740766 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:40.740795 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:40.740799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:40.741182 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:40.742205 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:40.742216 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:40.742677 (+ 461us) client_negotiation.cc:769] Sending connection context
0812 16:45:40.743050 (+ 373us) client_negotiation.cc:240] Negotiation successful
0812 16:45:40.743705 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":382,"threads_started":1}
I20260812 16:45:40.747398 12064 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42539
I20260812 16:45:40.747830 12064 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:40.749018 12064 heartbeater.cc:511] Master 127.28.243.62:42539 requested a full tablet report, sending...
I20260812 16:45:40.751829 11546 ts_manager.cc:194] Registered new tserver with Master: b83e3dde9d9e4d348afadbe2e7a05dbe (127.28.243.2:43111)
I20260812 16:45:40.752758 11546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:44581
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:45:40.808852 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42539
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:41.154676 11844 heartbeater.cc:503] Master 127.28.243.62:42539 was elected leader, sending a full tablet report...
W20260812 16:45:41.260316 12071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:41.260632 12071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:41.260735 12071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:41.269228 12071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:41.269371 12071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:41.269488 12071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:41.282615 12071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:41.284125 12071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:41.285543 12071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:41.300047 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:41.300083 12077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:41.299976 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:41.301029 12071 server_base.cc:1034] running on GCE node
I20260812 16:45:41.301702 12071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:41.302814 12071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:41.304086 12071 hybrid_clock.cc:648] HybridClock initialized: now 1786553141304022 us; error 62 us; skew 500 ppm
I20260812 16:45:41.304486 12071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:41 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553141, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:41.308768 12071 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:45:41.311717 12071 webserver.cc:460] Webserver started at http://127.28.243.3:40453/ using document root <none> and password file <none>
I20260812 16:45:41.313113 12071 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:41.313331 12071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:41.313777 12071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:45:41.319147 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "c1431503032c416192a27fdaebcbd385"
format_stamp: "Formatted at 2026-08-12 16:45:41 on dist-test-slave-1vmg"
server_key: "c43a5b9b53e32d9b7d779918af00079c"
server_key_iv: "c8a704a829f3b946bbebd77802fe06c6"
server_key_version: "encryptionkey@0"
I20260812 16:45:41.320310 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "c1431503032c416192a27fdaebcbd385"
format_stamp: "Formatted at 2026-08-12 16:45:41 on dist-test-slave-1vmg"
server_key: "c43a5b9b53e32d9b7d779918af00079c"
server_key_iv: "c8a704a829f3b946bbebd77802fe06c6"
server_key_version: "encryptionkey@0"
I20260812 16:45:41.329821 12071 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260812 16:45:41.338093 12087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:41.340433 12071 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260812 16:45:41.340729 12071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "c1431503032c416192a27fdaebcbd385"
format_stamp: "Formatted at 2026-08-12 16:45:41 on dist-test-slave-1vmg"
server_key: "c43a5b9b53e32d9b7d779918af00079c"
server_key_iv: "c8a704a829f3b946bbebd77802fe06c6"
server_key_version: "encryptionkey@0"
I20260812 16:45:41.341035 12071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:41.368211 12071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:41.374788 12071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:41.375293 12071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:41.376619 12071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:41.378327 12071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:41.378422 12071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:41.378553 12071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:41.378612 12071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:41.419883 12071 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:40765
I20260812 16:45:41.419903 12285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:40765 every 8 connection(s)
I20260812 16:45:41.421375 12071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:41.430215 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 12071
I20260812 16:45:41.430416 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:45:41.431349 29644 external_mini_cluster.cc:1428] Setting key ee1071b179c907b1575db332852a2db6
Aug 12 16:45:41 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553141, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:45:41.449558 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:41.424491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:53291 (local address 127.28.243.62:42539)
0812 16:45:41.425014 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:41.425019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:41.426182 (+ 1163us) server_negotiation.cc:407] Connection header received
0812 16:45:41.428471 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:41.428478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:41.428630 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:41.428794 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:41.431648 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.432906 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.435718 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.436356 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.436483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:41.441225 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:41.441234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:41.441238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:41.441300 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:41.444410 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.445480 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.445486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.445488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.445643 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.446155 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.446159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.446161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.446543 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:41.446677 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:41.448213 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0812 16:45:41.448731 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":272,"threads_started":1}
I20260812 16:45:41.451020 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:41.425121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:53291)
0812 16:45:41.425971 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:41.426029 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:41.427965 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:41.429151 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:41.429162 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:41.429844 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:41.431249 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:41.431309 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.433291 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.433298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:41.435468 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:41.435493 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.436659 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.436666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:41.437160 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:41.437956 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:41.437992 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:41.440707 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:45:41.444665 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:41.444677 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:41.444695 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:41.445264 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:41.445798 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:45:41.445802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:45:41.445805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:45:41.446028 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:45:41.446694 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:45:41.446709 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:45:41.447531 (+ 822us) client_negotiation.cc:769] Sending connection context
0812 16:45:41.447912 (+ 381us) client_negotiation.cc:240] Negotiation successful
0812 16:45:41.449486 (+ 1574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":332,"threads_started":1}
I20260812 16:45:41.452595 12286 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42539
I20260812 16:45:41.452934 12286 heartbeater.cc:464] Registering TS with master...
I20260812 16:45:41.454468 12286 heartbeater.cc:511] Master 127.28.243.62:42539 requested a full tablet report, sending...
I20260812 16:45:41.456888 11546 ts_manager.cc:194] Registered new tserver with Master: c1431503032c416192a27fdaebcbd385 (127.28.243.3:40765)
I20260812 16:45:41.458005 11546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:53291
I20260812 16:45:41.463963 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab.
Entry for principal oryx/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab.
WARNING: no policy specified for oryx/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab.
Entry for principal oryx/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab.
WARNING: no policy specified for oryx/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab.
Entry for principal oryx/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab.
I20260812 16:45:41.687259 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 11642
Aug 12 16:45:41 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:45:41.708702 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:41.688868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.28.243.1:32881)
0812 16:45:41.690099 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:41.690114 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:41.690368 (+ 254us) server_negotiation.cc:407] Connection header received
0812 16:45:41.690466 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:41.690473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:41.690715 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:41.691129 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:41.692980 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.694211 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.696405 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.697102 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.697243 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:41.701665 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:41.701684 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:41.701704 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:41.701752 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:41.704611 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.705430 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.705445 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.705448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.705613 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.706190 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.706197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.706200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.706841 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:41.707323 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:41.707736 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 16:45:41.708166 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":605,"threads_started":1}
I20260812 16:45:41.710588 11837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:41.756626 12064 heartbeater.cc:503] Master 127.28.243.62:42539 was elected leader, sending a full tablet report...
I20260812 16:45:41.807070 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 11642
I20260812 16:45:41.840174 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 11853
Aug 12 16:45:41 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:45:41.860939 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:41.841252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.28.243.2:43111)
0812 16:45:41.841930 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:41.841937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:41.841964 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:45:41.842029 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:41.842035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:41.842264 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:41.842400 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:41.843776 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.844861 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.846450 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.847549 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.847739 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:41.853048 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:41.853067 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:41.853087 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:41.853136 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:41.857574 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.858481 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.858490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.858492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.858658 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.859185 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.859191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.859211 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.859647 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:41.859945 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:41.860271 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 16:45:41.860523 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260812 16:45:41.862358 12056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:41.950644 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 11853
I20260812 16:45:41.978549 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12071
Aug 12 16:45:41 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:45:42.001801 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:41.979944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.28.243.3:40765)
0812 16:45:41.980824 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:41.980834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:41.980976 (+ 142us) server_negotiation.cc:407] Connection header received
0812 16:45:41.981144 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:41.981154 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:41.981419 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:41.981609 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:41.983382 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.984528 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.986120 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:41.987125 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:41.987279 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:41.993694 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:41.993716 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:41.993735 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:41.993791 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:41.997563 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.998624 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.998633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.998636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:41.998808 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:41.999327 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:41.999335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:41.999337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:42.000213 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:42.000612 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:42.001097 (+ 485us) server_negotiation.cc:299] Negotiation successful
0812 16:45:42.001430 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":491,"threads_started":1}
I20260812 16:45:42.003333 12278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:42.093103 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 12071
I20260812 16:45:42.114739 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 11496
I20260812 16:45:42.116942 11594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:42.237840 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 11496
I20260812 16:45:42.270418 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42539
--webserver_interface=127.28.243.62
--webserver_port=35453
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:45:42.653445 12316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:42.653779 12316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:42.653923 12316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:42.663414 12316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:45:42.663556 12316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:42.663609 12316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:42.663693 12316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:45:42.663757 12316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:45:42.675429 12316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:42539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=35453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:42.676959 12316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:42.678972 12316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:42.694774 12326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:42.694944 12322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:42.694824 12324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:42.696147 12316 server_base.cc:1034] running on GCE node
I20260812 16:45:42.697098 12316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:42.698740 12316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:42.699983 12316 hybrid_clock.cc:648] HybridClock initialized: now 1786553142699914 us; error 80 us; skew 500 ppm
I20260812 16:45:42.700424 12316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:42 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:42.703910 12316 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:45:42.705686 12316 webserver.cc:460] Webserver started at http://127.28.243.62:35453/ using document root <none> and password file <none>
I20260812 16:45:42.706502 12316 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:42.706650 12316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:42.712957 12316 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:45:42.716017 12341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:42.717659 12316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:45:42.717868 12316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "4913db1e283e444ea8f19d40b279c060"
format_stamp: "Formatted at 2026-08-12 16:45:39 on dist-test-slave-1vmg"
server_key: "769610e056a7241990fb27c4d4ab6212"
server_key_iv: "8874bb1579efee4aa8348eb77b5d29f7"
server_key_version: "encryptionkey@0"
I20260812 16:45:42.718969 12316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:42.737208 12316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:42.743134 12316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:42.743422 12316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:42.765808 12316 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:42539
I20260812 16:45:42.766016 12417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:42539 every 8 connection(s)
I20260812 16:45:42.767750 12316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:45:42.775372 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 12316
I20260812 16:45:42.776098 12418 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:45:42.776412 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:32881
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:42539
--webserver_port=46771
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:45:42.780972 12418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Bootstrap starting.
I20260812 16:45:42.808741 12418 log.cc:824] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Log is configured to *not* fsync() on all Append() calls
I20260812 16:45:42.822296 12418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:45:42.823638 12418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Bootstrap complete.
I20260812 16:45:42.835175 12418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:42.836531 12418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4913db1e283e444ea8f19d40b279c060, State: Initialized, Role: FOLLOWER
I20260812 16:45:42.837808 12418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:42.838199 12418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:45:42.838349 12418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:45:42.838570 12418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:45:42.843317 12418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:42.843954 12418 leader_election.cc:302] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4913db1e283e444ea8f19d40b279c060; no voters:
I20260812 16:45:42.844622 12418 leader_election.cc:288] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:45:42.844780 12425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:45:42.846031 12425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [term 2 LEADER]: Becoming Leader. State: Replica: 4913db1e283e444ea8f19d40b279c060, State: Running, Role: LEADER
I20260812 16:45:42.846793 12425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } }
I20260812 16:45:42.847394 12418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:45:42.849980 12427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4913db1e283e444ea8f19d40b279c060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } } }
I20260812 16:45:42.850459 12427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:42.849951 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4913db1e283e444ea8f19d40b279c060. Latest consensus state: current_term: 2 leader_uuid: "4913db1e283e444ea8f19d40b279c060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4913db1e283e444ea8f19d40b279c060" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42539 } } }
I20260812 16:45:42.852402 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060 [sys.catalog]: This master's current role is: LEADER
I20260812 16:45:42.863268 12433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:45:42.870352 12433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:45:42.872004 12433 catalog_manager.cc:1259] Loaded cluster ID: d1bfbc2e58a74499875ce31dee356aeb
I20260812 16:45:42.872146 12433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:45:42.876509 12433 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:45:42.879820 12433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4913db1e283e444ea8f19d40b279c060: Loaded TSK: 0
I20260812 16:45:42.882475 12433 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:45:43.186026 12421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:43.186388 12421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:43.186533 12421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:43.194463 12421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:43.194628 12421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:43.194792 12421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:45:43.206449 12421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.1.keytab
--principal=oryx/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:32881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=46771
--tserver_master_addrs=127.28.243.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:43.208082 12421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:43.209537 12421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:43.225571 12459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:43.225589 12463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:43.228147 12465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:43.228948 12421 server_base.cc:1034] running on GCE node
I20260812 16:45:43.229642 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:43.230696 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:43.231930 12421 hybrid_clock.cc:648] HybridClock initialized: now 1786553143231879 us; error 75 us; skew 500 ppm
I20260812 16:45:43.232272 12421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:43 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553143, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:43.236073 12421 init.cc:377] Logged in from keytab as oryx/127.28.243.1@KRBTEST.COM (short username oryx)
I20260812 16:45:43.237920 12421 webserver.cc:460] Webserver started at http://127.28.243.1:46771/ using document root <none> and password file <none>
I20260812 16:45:43.239243 12421 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:43.239429 12421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:43.246376 12421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 16:45:43.249760 12483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:43.251382 12421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:45:43.251574 12421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "edfb7a4465af4ac18d12452ca8d1f998"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "d2f337d35c788446f04ffea0044c4f05"
server_key_iv: "3338620e637a5f5f0b18381d23f44c45"
server_key_version: "encryptionkey@0"
I20260812 16:45:43.252638 12421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:43.273974 12421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:43.278949 12421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:43.279251 12421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:43.280736 12421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:43.282641 12421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:43.282758 12421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:43.282873 12421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:43.282979 12421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 16:45:43.317507 12421 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:32881
I20260812 16:45:43.317579 12668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:32881 every 8 connection(s)
I20260812 16:45:43.319199 12421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:45:43.326371 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 12421
I20260812 16:45:43.327608 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:43111
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:42539
--webserver_port=40881
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:45:43 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:43 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 16:45:43.721347 12680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:43.721671 12680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:43.721781 12680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:43.729496 12680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:43.729631 12680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:43.729776 12680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:45:43.741133 12680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.2.keytab
--principal=oryx/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:43111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=40881
--tserver_master_addrs=127.28.243.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:43.742664 12680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:43.744163 12680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:43.764954 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:43.765158 12692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:43.765727 12694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:43.775059 12680 server_base.cc:1034] running on GCE node
I20260812 16:45:43.775965 12680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:43.777370 12680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:43.780831 12680 hybrid_clock.cc:648] HybridClock initialized: now 1786553143780731 us; error 94 us; skew 500 ppm
I20260812 16:45:43.781303 12680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:43 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553143, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:43.786319 12680 init.cc:377] Logged in from keytab as oryx/127.28.243.2@KRBTEST.COM (short username oryx)
I20260812 16:45:43.788645 12680 webserver.cc:460] Webserver started at http://127.28.243.2:40881/ using document root <none> and password file <none>
I20260812 16:45:43.789463 12680 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:43.789630 12680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:43.797528 12680 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:45:43.800850 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:43.802429 12680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:45:43.802613 12680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "b83e3dde9d9e4d348afadbe2e7a05dbe"
format_stamp: "Formatted at 2026-08-12 16:45:40 on dist-test-slave-1vmg"
server_key: "bd1389303fcc83e0888b9209314e5c79"
server_key_iv: "ade8c14cdc452b5285e1d6f01c10b52d"
server_key_version: "encryptionkey@0"
I20260812 16:45:43.803730 12680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:43.820464 12680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:43.825502 12680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:43.825803 12680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:43.826982 12680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:43.828572 12680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:43.828684 12680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:43.828802 12680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:43.828873 12680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:43.885386 12680 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:43111
I20260812 16:45:43.885442 12891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:43111 every 8 connection(s)
I20260812 16:45:43.887172 12680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:45:43.896469 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 12680
I20260812 16:45:43.897207 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:40765
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:42539
--webserver_port=40453
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:45:43 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:43 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 16:45:44.299710 12901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:45:44.300143 12901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:45:44.300320 12901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:45:44.308374 12901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:45:44.308531 12901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:45:44.308826 12901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:45:44.322258 12901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/krb5kdc/oryx_127.28.243.3.keytab
--principal=oryx/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:40765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=40453
--tserver_master_addrs=127.28.243.62:42539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:45:44.324980 12901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:45:44.327415 12901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:45:44.344707 12908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:44.345001 12909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:45:44.349414 12914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:45:44.350217 12901 server_base.cc:1034] running on GCE node
I20260812 16:45:44.351333 12901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:45:44.356288 12901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:45:44.359797 12901 hybrid_clock.cc:648] HybridClock initialized: now 1786553144359681 us; error 95 us; skew 500 ppm
I20260812 16:45:44.360430 12901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553144, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:45:44.376174 12901 init.cc:377] Logged in from keytab as oryx/127.28.243.3@KRBTEST.COM (short username oryx)
I20260812 16:45:44.380180 12901 webserver.cc:460] Webserver started at http://127.28.243.3:40453/ using document root <none> and password file <none>
I20260812 16:45:44.381579 12901 fs_manager.cc:359] Metadata directory not provided
I20260812 16:45:44.381832 12901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:45:44.391573 12901 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:45:44.396185 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:44.398634 12901 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:45:44.399047 12901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "c1431503032c416192a27fdaebcbd385"
format_stamp: "Formatted at 2026-08-12 16:45:41 on dist-test-slave-1vmg"
server_key: "c43a5b9b53e32d9b7d779918af00079c"
server_key_iv: "c8a704a829f3b946bbebd77802fe06c6"
server_key_version: "encryptionkey@0"
I20260812 16:45:44.400799 12901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:45:44.430749 12901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:45:44.436646 12901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:45:44.436961 12901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:45:44.438200 12901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:45:44.440104 12901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:45:44.440703 12901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:44.440987 12901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:45:44.441083 12901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:45:44.495610 12901 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:40765
I20260812 16:45:44.495663 13097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:40765 every 8 connection(s)
I20260812 16:45:44.497299 12901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:45:44.501948 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 12901
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:44.540444 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:44.515386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.28.243.62:42539)
0812 16:45:44.516000 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:44.516006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:44.516030 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:45:44.516099 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:44.516104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:44.516205 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:44.517401 (+ 1196us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:44.519043 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.520728 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.522383 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.523536 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.523770 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.528752 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:44.528765 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:44.528798 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:44.528862 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:44.534568 (+ 5706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:44.535685 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:44.535695 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:44.535704 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:44.535918 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:45:44.537482 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:45:44.537487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:45:44.537489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:45:44.538116 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:45:44.538446 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 16:45:44.539002 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 16:45:44.539547 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
W20260812 16:45:44.692127 12674 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.356s user 0.004s sys 0.013s
W20260812 16:45:44.694388 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:43.322581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:37415 (local address 127.28.243.62:42539)
0812 16:45:43.323385 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:43.323402 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:43.323609 (+ 207us) server_negotiation.cc:407] Connection header received
0812 16:45:43.326039 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:43.326058 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:43.326529 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:43.326988 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:43.329910 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.332015 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.334160 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.334876 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.335124 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.693970 (+1358846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:37415: BlockingRecv error: recv got EOF from 127.28.243.1:37415 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":335,"threads_started":1}
I20260812 16:45:44.695139 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:43.322445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:37415)
0812 16:45:43.323415 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:43.323472 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:43.325303 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:43.327806 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:43.327820 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:43.328755 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:43.329594 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:43.329622 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.332281 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.332287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:43.333958 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:43.333979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.335190 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.335199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:43.335482 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:43.336196 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:43.336226 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:44.693548 (+1357322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":391,"threads_started":1}
W20260812 16:45:44.696852 12669 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:42539 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.243.62:42539: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:44.714795 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:44.697942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:47601)
0812 16:45:44.698432 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:44.698461 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:44.698636 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:44.699801 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:44.699809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:44.700426 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:44.700703 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:44.700717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.702519 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.702535 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:44.704462 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:44.704488 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.705880 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.705889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:44.706126 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.707125 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:44.707159 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:44.714383 (+ 7224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299}
W20260812 16:45:44.715174 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:44.698655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:47601 (local address 127.28.243.62:42539)
0812 16:45:44.698876 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:44.698883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:44.698949 (+ 66us) server_negotiation.cc:407] Connection header received
0812 16:45:44.699044 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:44.699051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:44.699156 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:44.699474 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:44.700930 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.702260 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.704698 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.705587 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.705779 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.714804 (+ 9025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:47601: BlockingRecv error: recv got EOF from 127.28.243.1:47601 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:44 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:44.735387 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:44.716319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:58111)
0812 16:45:44.717281 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:44.717308 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:44.717521 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:44.717988 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:44.717994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:44.718820 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:44.719341 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:44.719359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.721327 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.721334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:44.724594 (+ 3260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:44.724632 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.727160 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.727174 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:44.727608 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.728706 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:44.728737 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:44.734904 (+ 6167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":457,"threads_started":1}
W20260812 16:45:44.735677 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:44.716390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:58111 (local address 127.28.243.62:42539)
0812 16:45:44.716633 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:44.716641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:44.717465 (+ 824us) server_negotiation.cc:407] Connection header received
0812 16:45:44.717694 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:44.717702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:44.717811 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:44.717934 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:44.719661 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.721108 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.724876 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.725744 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.725936 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.735313 (+ 9377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:58111: BlockingRecv error: recv got EOF from 127.28.243.1:58111 (error 108)
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":106}
W20260812 16:45:44.736414 12669 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:45:45.087589 12900 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.183s user 0.011s sys 0.008s
W20260812 16:45:45.089324 12897 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:43.890421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:50135 (local address 127.28.243.62:42539)
0812 16:45:43.893383 (+ 2962us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:43.893391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:43.893424 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:45:43.894747 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:43.894755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:43.894860 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:43.895034 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:43.897053 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.898762 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.901449 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.902343 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.902545 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.088915 (+1186370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:50135: BlockingRecv error: recv got EOF from 127.28.243.2:50135 (error 108)
Metrics: {"server-negotiator.queue_time_us":2806,"thread_start_us":2639,"threads_started":1}
I20260812 16:45:45.089529 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:43.891282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:50135)
0812 16:45:43.892399 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:43.892474 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:43.894428 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:43.895258 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:43.895268 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:43.895857 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:43.896758 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:43.896787 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.899110 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.899117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:43.901193 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:43.901216 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:43.902719 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:43.902727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:43.903118 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:43.904140 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:43.904172 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.088592 (+1184420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":408,"threads_started":1}
W20260812 16:45:45.090694 12893 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:42539 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.243.62:42539: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:45.101352 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:45.091552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:57557)
0812 16:45:45.091774 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:45.091797 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:45.091949 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:45.092271 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:45.092276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:45.092668 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:45.092841 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.092850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.093852 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.093858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.094832 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.094845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.095593 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.095600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.095753 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.096461 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:45.096478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.100936 (+ 4458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 16:45:45.101639 12897 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:45.091624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:57557 (local address 127.28.243.62:42539)
0812 16:45:45.091799 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:45.091809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:45.091872 (+ 63us) server_negotiation.cc:407] Connection header received
0812 16:45:45.092070 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:45.092075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:45.092143 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:45.092227 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:45.092987 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.093700 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.094993 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.095413 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.095527 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.101266 (+ 5739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:57557: BlockingRecv error: recv got EOF from 127.28.243.2:57557 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:45.113080 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:45.102834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:55927)
0812 16:45:45.103184 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:45.103202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:45.103329 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:45.103803 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:45.103809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:45.104357 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:45.104591 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.104605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.105878 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.105883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.107060 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.107076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.107841 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.107847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.108142 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.108931 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:45.108959 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.112813 (+ 3854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225}
W20260812 16:45:45.113461 12893 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:45:45.114177 12897 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:45.103228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:55927 (local address 127.28.243.62:42539)
0812 16:45:45.103494 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:45.103500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:45.103517 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:45:45.103571 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:45.103575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:45.103641 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:45.103731 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:45.104845 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.105692 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.107232 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.107660 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.107788 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.113950 (+ 6162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:55927: BlockingRecv error: recv got EOF from 127.28.243.2:55927 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260812 16:45:45.593003 13103 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.063s user 0.008s sys 0.015s
W20260812 16:45:45.594297 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:44.501296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:47677 (local address 127.28.243.62:42539)
0812 16:45:44.515442 (+ 14146us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:44.515448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:44.515474 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:45:44.518172 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:44.518185 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:44.518398 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:44.518646 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:44.520848 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.523825 (+ 2977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.527194 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.528041 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.528265 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.594027 (+1065762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:47677: BlockingRecv error: recv got EOF from 127.28.243.3:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":13981,"thread_start_us":13818,"threads_started":1}
I20260812 16:45:45.594959 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:44.502154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:47677)
0812 16:45:44.514620 (+ 12466us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:44.514696 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:44.517505 (+ 2809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:44.519049 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:44.519061 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:44.519776 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:44.520554 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:44.520576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.524110 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.524116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:44.526861 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:44.526888 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:44.528409 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:44.528417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:44.528671 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:44.529447 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:44.529473 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.593829 (+1064356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12062,"thread_start_us":11604,"threads_started":1}
W20260812 16:45:45.595955 13099 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:42539 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.243.62:42539: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:45.607360 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:45.596632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:49817)
0812 16:45:45.596843 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:45.596861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:45.596995 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:45.597356 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:45.597362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:45.597741 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:45.597914 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.597923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.599753 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.599772 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.600796 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.600812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.601630 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.601637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.601799 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.602491 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:45.602512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.606999 (+ 4487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 16:45:45.607503 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:45.596877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:49817 (local address 127.28.243.62:42539)
0812 16:45:45.597079 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:45.597084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:45.597102 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:45:45.597155 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:45.597159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:45.597222 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:45.597304 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:45.598383 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.599543 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.600983 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.601423 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.601542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.607269 (+ 5727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:49817: BlockingRecv error: recv got EOF from 127.28.243.3:49817 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:45.617978 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:45.608206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:47501)
0812 16:45:45.608385 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:45.608401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:45.608523 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:45.608917 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:45.608923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:45.609319 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:45.609500 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.609510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.610543 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.610550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.611740 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.611756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.612547 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.612554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.612748 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.613484 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:45.613505 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.617705 (+ 4200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 16:45:45.618265 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:45.608331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:47501 (local address 127.28.243.62:42539)
0812 16:45:45.608629 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:45.608635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:45.608652 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:45:45.608702 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:45.608706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:45.608764 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:45.608847 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:45.609648 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.610398 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.611904 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.612372 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.612514 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.617987 (+ 5473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:47501: BlockingRecv error: recv got EOF from 127.28.243.3:47501 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
W20260812 16:45:45.618402 13099 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:45 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:45.748729 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:45.737056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:47953)
0812 16:45:45.737593 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:45.737610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:45.737737 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:45.738188 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:45.738194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:45.738699 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:45.738990 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.739002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.740359 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.740366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.741459 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:45.741481 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.742338 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.742343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:45.742491 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.743269 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:45.743298 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:45.748404 (+ 5106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":308,"threads_started":1}
W20260812 16:45:45.748920 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:45.737290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:47953 (local address 127.28.243.62:42539)
0812 16:45:45.737608 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:45.737613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:45.737674 (+ 61us) server_negotiation.cc:407] Connection header received
0812 16:45:45.737833 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:45.737837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:45.737907 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:45.738006 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:45.739235 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.740116 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.741687 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:45.742136 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:45.742248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:45.748698 (+ 6450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:47953: BlockingRecv error: recv got EOF from 127.28.243.1:47953 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 12 16:45:46 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:46 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:46.126971 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:46.115065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:58687)
0812 16:45:46.115882 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:46.115903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:46.116109 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:46.116522 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:46.116528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:46.117172 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:46.117421 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:46.117433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.118980 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.118985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:46.120377 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:46.120408 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.121351 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.121358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:46.121521 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:46.122434 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:46.122455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:46.126635 (+ 4180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":469,"threads_started":1}
W20260812 16:45:46.127631 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:46.115348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:58687 (local address 127.28.243.62:42539)
0812 16:45:46.115710 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:46.115718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:46.116027 (+ 309us) server_negotiation.cc:407] Connection header received
0812 16:45:46.116268 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:46.116272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:46.116357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:46.116456 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:46.117638 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.118753 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.120643 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.121103 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.121235 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:46.127325 (+ 6090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:58687: BlockingRecv error: recv got EOF from 127.28.243.2:58687 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Aug 12 16:45:46 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:46 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:46.629444 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:46.619319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:53561)
0812 16:45:46.620144 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:46.620161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:46.620304 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:46.620652 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:46.620658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:46.621134 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:46.621294 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:46.621303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.622322 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.622328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:46.623366 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:46.623380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.624197 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.624203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:46.624363 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:46.624945 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:46.624962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:46.628872 (+ 3910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":561,"threads_started":1}
W20260812 16:45:46.629488 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:46.619439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:53561 (local address 127.28.243.62:42539)
0812 16:45:46.619647 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:46.619653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:46.620254 (+ 601us) server_negotiation.cc:407] Connection header received
0812 16:45:46.620426 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:46.620432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:46.620505 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:46.620607 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:46.621445 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.622183 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.623536 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.623996 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.624116 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:46.629254 (+ 5138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:53561: BlockingRecv error: recv got EOF from 127.28.243.3:53561 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 16:45:46 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:46 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:46.763283 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:46.750666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:44925)
0812 16:45:46.751413 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:46.751438 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:46.751745 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:46.752238 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:46.752243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:46.752992 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:46.753321 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:46.753338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.754754 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.754760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:46.756257 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:46.756277 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.757411 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.757417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:46.757576 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:46.758478 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:46.758503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:46.763005 (+ 4502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":692,"threads_started":1}
W20260812 16:45:46.763733 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:46.750867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44925 (local address 127.28.243.62:42539)
0812 16:45:46.751229 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:46.751236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:46.751662 (+ 426us) server_negotiation.cc:407] Connection header received
0812 16:45:46.751883 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:46.751887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:46.752044 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:46.752174 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:46.753530 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.754567 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.756481 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:46.757203 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:46.757347 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:46.763333 (+ 5986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:44925: BlockingRecv error: recv got EOF from 127.28.243.1:44925 (error 108)
Metrics: {"server-negotiator.queue_time_us":193}
Aug 12 16:45:47 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:47 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:47.140952 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:47.128497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:56605)
0812 16:45:47.129261 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:47.129284 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:47.129519 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:47.130202 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:47.130212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:47.131203 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:47.131431 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:47.131444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.132744 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.132751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:47.134032 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:47.134132 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.135191 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.135198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:47.135411 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:47.136370 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:47.136397 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:47.140653 (+ 4256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":439,"threads_started":1}
W20260812 16:45:47.141176 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:47.128607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:56605 (local address 127.28.243.62:42539)
0812 16:45:47.129058 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:47.129067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:47.129437 (+ 370us) server_negotiation.cc:407] Connection header received
0812 16:45:47.129675 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:47.129680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:47.129787 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:47.130054 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:47.131653 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.132425 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.134378 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.134982 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.135132 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:47.140932 (+ 5800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:56605: BlockingRecv error: recv got EOF from 127.28.243.2:56605 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 16:45:47 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:47 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:47.643224 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:47.631264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:32913)
0812 16:45:47.632329 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:47.632349 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:47.632867 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:47.633310 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:47.633315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:47.634000 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:47.634261 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:47.634278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.635534 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.635538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:47.636500 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:47.636515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.637449 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.637455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:47.637646 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:47.638343 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:47.638364 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:47.642887 (+ 4523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":796,"threads_started":1}
W20260812 16:45:47.643520 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:47.631500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:32913 (local address 127.28.243.62:42539)
0812 16:45:47.631739 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:47.631746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:47.632484 (+ 738us) server_negotiation.cc:407] Connection header received
0812 16:45:47.633053 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:47.633058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:47.633148 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:47.633246 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:47.634491 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.635346 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.636683 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.637237 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.637381 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:47.643224 (+ 5843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:32913: BlockingRecv error: recv got EOF from 127.28.243.3:32913 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 12 16:45:47 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:47 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:47.776664 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:47.764504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:54943)
0812 16:45:47.765133 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:47.765151 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:47.765305 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:47.765764 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:47.765769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:47.766364 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:47.766552 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:47.766562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.767697 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.767703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:47.768813 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:47.768830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.769641 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.769648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:47.769830 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:47.770710 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:47.770744 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:47.776405 (+ 5661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":345,"threads_started":1}
W20260812 16:45:47.777288 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:47.764648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:54943 (local address 127.28.243.62:42539)
0812 16:45:47.764927 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:47.764933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:47.765242 (+ 309us) server_negotiation.cc:407] Connection header received
0812 16:45:47.765467 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:47.765471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:47.765570 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:47.765695 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:47.766704 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.767535 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.768991 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:47.769487 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:47.769604 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:47.776781 (+ 7177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:54943: BlockingRecv error: recv got EOF from 127.28.243.1:54943 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 12 16:45:48 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:48 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:48.155328 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:48.142339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:55937)
0812 16:45:48.142883 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:48.142900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:48.143061 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:48.143442 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:48.143445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:48.143937 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:48.144168 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:48.144179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.145474 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.145478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:48.146639 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:48.146655 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.147503 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.147509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:48.147832 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:48.148850 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:48.148896 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:48.155026 (+ 6130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":270,"threads_started":1}
W20260812 16:45:48.155895 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:48.142509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:55937 (local address 127.28.243.62:42539)
0812 16:45:48.142713 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:48.142718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:48.143053 (+ 335us) server_negotiation.cc:407] Connection header received
0812 16:45:48.143229 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:48.143234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:48.143308 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:48.143414 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:48.144364 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.145310 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.146782 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.147309 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.147414 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:48.155646 (+ 8232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:55937: BlockingRecv error: recv got EOF from 127.28.243.2:55937 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 16:45:48 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:48 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:48.656437 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:48.644677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:45927)
0812 16:45:48.645258 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:48.645275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:48.645430 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:48.645801 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:48.645806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:48.646249 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:48.646420 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:48.646429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.647587 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.647591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:48.648643 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:48.648659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.649402 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.649409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:48.649589 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:48.650566 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:48.650604 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:48.656118 (+ 5514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":295,"threads_started":1}
W20260812 16:45:48.656787 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:48.644919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:45927 (local address 127.28.243.62:42539)
0812 16:45:48.645167 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:48.645175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:48.645358 (+ 183us) server_negotiation.cc:407] Connection header received
0812 16:45:48.645569 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:48.645574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:48.645661 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:48.645751 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:48.646574 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.647427 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.648793 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.649242 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.649360 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:48.656519 (+ 7159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:45927: BlockingRecv error: recv got EOF from 127.28.243.3:45927 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 12 16:45:48 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:48 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:48.789508 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:48.778075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:44633)
0812 16:45:48.778622 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:48.778641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:48.778774 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:48.779223 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:48.779226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:48.779649 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:48.779834 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:48.779844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.781187 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.781195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:48.782432 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:48.782454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.783447 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.783453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:48.783655 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:48.784407 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:48.784432 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:48.789138 (+ 4706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":308,"threads_started":1}
W20260812 16:45:48.789960 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:48.778251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44633 (local address 127.28.243.62:42539)
0812 16:45:48.778470 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:48.778475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:48.778742 (+ 267us) server_negotiation.cc:407] Connection header received
0812 16:45:48.778889 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:48.778894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:48.779007 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:48.779114 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:48.780079 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.780854 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.782699 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:48.783240 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:48.783378 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:48.789424 (+ 6046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:44633: BlockingRecv error: recv got EOF from 127.28.243.1:44633 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 12 16:45:49 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:49 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:49.168313 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:49.156745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:54953)
0812 16:45:49.157531 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:49.157556 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:49.157720 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:49.158146 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:49.158152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:49.158697 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:49.158970 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:49.158986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.160566 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.160572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:49.161654 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:49.161675 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.162549 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.162555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:49.162748 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:49.163497 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:49.163517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:49.167913 (+ 4396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":454,"threads_started":1}
W20260812 16:45:49.168460 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:49.156901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:54953 (local address 127.28.243.62:42539)
0812 16:45:49.157106 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:49.157114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:49.157656 (+ 542us) server_negotiation.cc:407] Connection header received
0812 16:45:49.157861 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:49.157867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:49.157974 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:49.158067 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:49.159224 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.160266 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.161866 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.162363 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.162495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:49.168228 (+ 5733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:54953: BlockingRecv error: recv got EOF from 127.28.243.2:54953 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 16:45:49 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:49 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:49.669637 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:49.658080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:46529)
0812 16:45:49.658756 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:49.658777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:49.658976 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:49.659479 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:49.659483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:49.660061 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:49.660280 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:49.660291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.661518 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.661524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:49.662629 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:49.662647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.663645 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.663651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:49.663807 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:49.664629 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:49.664650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:49.669348 (+ 4698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":353,"threads_started":1}
W20260812 16:45:49.669898 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:49.658161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:46529 (local address 127.28.243.62:42539)
0812 16:45:49.658432 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:49.658440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:49.658894 (+ 454us) server_negotiation.cc:407] Connection header received
0812 16:45:49.659147 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:49.659152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:49.659287 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:49.659403 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:49.660506 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.661277 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.662961 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.663438 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.663552 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:49.669635 (+ 6083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:46529: BlockingRecv error: recv got EOF from 127.28.243.3:46529 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 16:45:49 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:49 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:49.803696 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:49.791248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:53251)
0812 16:45:49.792185 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:49.792208 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:49.792388 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:49.792967 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:49.792972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:49.793573 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:49.793847 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:49.793912 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.795594 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.795599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:49.796804 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:49.796820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.797684 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.797689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:49.797854 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:49.798771 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:49.798826 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:49.803434 (+ 4608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":404,"threads_started":1}
W20260812 16:45:49.803975 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:49.791492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:53251 (local address 127.28.243.62:42539)
0812 16:45:49.791764 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:49.791773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:49.792377 (+ 604us) server_negotiation.cc:407] Connection header received
0812 16:45:49.792605 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:49.792625 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:49.792744 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:49.792870 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:49.794224 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.795352 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.797053 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:49.797485 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:49.797598 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:49.803742 (+ 6144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:53251: BlockingRecv error: recv got EOF from 127.28.243.1:53251 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 12 16:45:50 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:50 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:50.183055 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:50.169851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:51787)
0812 16:45:50.171463 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:50.171488 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:50.171709 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:50.172421 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:50.172430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:50.172964 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:50.173168 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:50.173179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.174514 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.174519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:50.175887 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:50.175917 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.176964 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.176978 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:50.177174 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:50.177923 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:50.177944 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:50.182728 (+ 4784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1432,"thread_start_us":1291,"threads_started":1}
W20260812 16:45:50.183328 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:50.170307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:51787 (local address 127.28.243.62:42539)
0812 16:45:50.170743 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:50.170750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:50.171638 (+ 888us) server_negotiation.cc:407] Connection header received
0812 16:45:50.172002 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:50.172010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:50.172120 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:50.172250 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:50.173368 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.174318 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.176170 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.176754 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.176912 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:50.183053 (+ 6141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:51787: BlockingRecv error: recv got EOF from 127.28.243.2:51787 (error 108)
Metrics: {"server-negotiator.queue_time_us":204}
Aug 12 16:45:50 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:50 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:50.682763 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:50.671580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:59923)
0812 16:45:50.672412 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:50.672431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:50.672627 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:50.673035 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:50.673041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:50.673637 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:50.673825 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:50.673836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.675076 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.675081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:50.676119 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:50.676134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.676914 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.676920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:50.677107 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:50.678167 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:50.678196 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:50.682403 (+ 4207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":500,"threads_started":1}
W20260812 16:45:50.683003 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:50.671609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:59923 (local address 127.28.243.62:42539)
0812 16:45:50.672073 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:50.672083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:50.672557 (+ 474us) server_negotiation.cc:407] Connection header received
0812 16:45:50.672758 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:50.672764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:50.672869 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:50.672976 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:50.674000 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.674877 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.676287 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.676746 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.676866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:50.682732 (+ 5866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:59923: BlockingRecv error: recv got EOF from 127.28.243.3:59923 (error 108)
Metrics: {"server-negotiator.queue_time_us":237}
Aug 12 16:45:50 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:50 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:50.817284 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:50.805112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:53657)
0812 16:45:50.805700 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:50.805718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:50.805864 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:50.806233 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:50.806236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:50.806725 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:50.806957 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:50.806967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.808272 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.808280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:50.809517 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:50.809544 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.810778 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.810784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:50.811007 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:50.811973 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:50.812010 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:50.816956 (+ 4946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":275,"threads_started":1}
W20260812 16:45:50.817731 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:50.805273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:53657 (local address 127.28.243.62:42539)
0812 16:45:50.805535 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:50.805543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:50.805814 (+ 271us) server_negotiation.cc:407] Connection header received
0812 16:45:50.806010 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:50.806015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:50.806088 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:50.806201 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:50.807138 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.807971 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.809789 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:50.810498 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:50.810656 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:50.817375 (+ 6719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:53657: BlockingRecv error: recv got EOF from 127.28.243.1:53657 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 16:45:51 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:51 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:51.195891 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:51.184299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:55417)
0812 16:45:51.184960 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:51.184979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:51.185135 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:51.185583 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:51.185588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:51.186082 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:51.186301 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:51.186313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.187337 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.187342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:51.188302 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:51.188315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.189465 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.189473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:51.189642 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:51.190403 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:51.190432 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:51.195515 (+ 5083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":409,"threads_started":1}
W20260812 16:45:51.197214 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:51.184496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:55417 (local address 127.28.243.62:42539)
0812 16:45:51.184719 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:51.184726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:51.185059 (+ 333us) server_negotiation.cc:407] Connection header received
0812 16:45:51.185287 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:51.185291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:51.185375 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:51.185481 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:51.186428 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.187173 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.188473 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.188927 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.189274 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:51.196113 (+ 6839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:55417: BlockingRecv error: recv got EOF from 127.28.243.2:55417 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 16:45:51 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:51 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:51.695986 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:51.684026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:44415)
0812 16:45:51.684551 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:51.684569 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:51.684712 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:51.685006 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:51.685009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:51.685639 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:51.685832 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:51.685844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.687044 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.687048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:51.688029 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:51.688050 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.688911 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.688918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:51.689204 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:51.690230 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:51.690260 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:51.695698 (+ 5438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":284,"threads_started":1}
W20260812 16:45:51.696288 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:51.684203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:44415 (local address 127.28.243.62:42539)
0812 16:45:51.684378 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:51.684383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:51.684740 (+ 357us) server_negotiation.cc:407] Connection header received
0812 16:45:51.684810 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:51.684814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:51.684879 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:51.684973 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:51.686028 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.686818 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.688232 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.688682 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.688795 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:51.696020 (+ 7225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:44415: BlockingRecv error: recv got EOF from 127.28.243.3:44415 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 16:45:51 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:51 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:51.828325 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:51.818638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:54727)
0812 16:45:51.819221 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:51.819238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:51.819381 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:51.819734 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:51.819738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:51.820160 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:51.820336 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:51.820344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.821435 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.821439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:51.822474 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:51.822489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.823426 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.823431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:51.823591 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:51.824293 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:51.824312 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:51.828035 (+ 3723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":273,"threads_started":1}
W20260812 16:45:51.828825 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:51.818776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:54727 (local address 127.28.243.62:42539)
0812 16:45:51.819070 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:51.819077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:51.819353 (+ 276us) server_negotiation.cc:407] Connection header received
0812 16:45:51.819496 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:51.819500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:51.819579 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:51.819684 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:51.820471 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.821243 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.822667 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:51.823258 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:51.823391 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:51.828631 (+ 5240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:54727: BlockingRecv error: recv got EOF from 127.28.243.1:54727 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 12 16:45:52 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:52 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:52.206938 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:52.197169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:55841)
0812 16:45:52.197722 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:52.197739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:52.197862 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:52.198146 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:52.198149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:52.198584 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:52.198765 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:52.198773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.199967 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.199971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:52.200938 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:52.200951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.201744 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.201749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:52.201878 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:52.202519 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:52.202537 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:52.206552 (+ 4015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":235,"threads_started":1}
W20260812 16:45:52.207109 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:52.197253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:55841 (local address 127.28.243.62:42539)
0812 16:45:52.197520 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:52.197526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:52.197820 (+ 294us) server_negotiation.cc:407] Connection header received
0812 16:45:52.197956 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:52.197960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:52.198024 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:52.198105 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:52.198984 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.199802 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.201093 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.201542 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.201644 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:52.206800 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:55841: BlockingRecv error: recv got EOF from 127.28.243.2:55841 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 12 16:45:52 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:52 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:52.708505 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:52.697861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:41367)
0812 16:45:52.698630 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:52.698662 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:52.698816 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:52.699168 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:52.699171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:52.699589 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:52.699795 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:52.699805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.700909 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.700914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:52.702204 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:52.702223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.703105 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.703111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:52.703245 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:52.703999 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:52.704021 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:52.707912 (+ 3891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":323,"threads_started":1}
W20260812 16:45:52.708796 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:52.698095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:41367 (local address 127.28.243.62:42539)
0812 16:45:52.698304 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:52.698312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:52.698761 (+ 449us) server_negotiation.cc:407] Connection header received
0812 16:45:52.698959 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:52.698965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:52.699037 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:52.699133 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:52.699956 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.700725 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.702421 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.702870 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.703039 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:52.708489 (+ 5450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:41367: BlockingRecv error: recv got EOF from 127.28.243.3:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 16:45:52 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:52 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:52.841153 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:52.829867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:53773)
0812 16:45:52.830563 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:52.830580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:52.830734 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:52.831113 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:52.831118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:52.831595 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:52.831792 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:52.831802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.832927 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.832933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:52.833935 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:52.833949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.834891 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.834898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:52.835229 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:52.835907 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:52.836122 (+ 215us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:52.840793 (+ 4671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":402,"threads_started":1}
W20260812 16:45:52.841269 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:52.830180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:53773 (local address 127.28.243.62:42539)
0812 16:45:52.830427 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:52.830434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:52.830678 (+ 244us) server_negotiation.cc:407] Connection header received
0812 16:45:52.830861 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:52.830867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:52.830963 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:52.831064 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:52.831955 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.832772 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.834111 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:52.834594 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:52.834723 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:52.841017 (+ 6294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:53773: BlockingRecv error: recv got EOF from 127.28.243.1:53773 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 16:45:53 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:53 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:53.219699 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:53.208345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:60451)
0812 16:45:53.208926 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:53.208944 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:53.209074 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:53.209414 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:53.209417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:53.209874 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:53.210075 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:53.210086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.211523 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.211528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:53.212611 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:53.212626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.213637 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.213647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:53.213894 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:53.214697 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:53.214721 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:53.219310 (+ 4589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":310,"threads_started":1}
W20260812 16:45:53.219964 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:53.208496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:60451 (local address 127.28.243.62:42539)
0812 16:45:53.208728 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:53.208733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:53.209036 (+ 303us) server_negotiation.cc:407] Connection header received
0812 16:45:53.209198 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:53.209203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:53.209272 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:53.209368 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:53.210256 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.211367 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.212756 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.213208 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.213528 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:53.219699 (+ 6171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:60451: BlockingRecv error: recv got EOF from 127.28.243.2:60451 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 16:45:53 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:53 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:53.721936 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:53.710237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:53377)
0812 16:45:53.710829 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:53.710846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:53.711022 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:53.711361 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:53.711366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:53.711837 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:53.712067 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:53.712081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.713462 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.713468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:53.715330 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:53.715359 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.716294 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.716302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:53.716471 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:53.717346 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:53.717367 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:53.721661 (+ 4294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":310,"threads_started":1}
W20260812 16:45:53.722203 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:53.710415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:53377 (local address 127.28.243.62:42539)
0812 16:45:53.710676 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:53.710683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:53.710944 (+ 261us) server_negotiation.cc:407] Connection header received
0812 16:45:53.711153 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:53.711158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:53.711223 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:53.711309 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:53.712228 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.713309 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.715548 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.716056 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.716193 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:53.721948 (+ 5755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:53377: BlockingRecv error: recv got EOF from 127.28.243.3:53377 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 12 16:45:53 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:53 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:53.851990 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:53.842328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:52609)
0812 16:45:53.842859 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:53.842881 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:53.843049 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:53.843411 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:53.843416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:53.843934 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:53.844097 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:53.844106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.845109 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.845115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:53.846051 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:53.846063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.846773 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.846779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:53.846977 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:53.847590 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:53.847618 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:53.851623 (+ 4005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
W20260812 16:45:53.852108 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:53.842388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:52609 (local address 127.28.243.62:42539)
0812 16:45:53.842628 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:53.842635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:53.843003 (+ 368us) server_negotiation.cc:407] Connection header received
0812 16:45:53.843194 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:53.843198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:53.843264 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:53.843355 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:53.844253 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.844973 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.846205 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:53.846612 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:53.846723 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:53.851861 (+ 5138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:52609: BlockingRecv error: recv got EOF from 127.28.243.1:52609 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 16:45:54 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:54 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:54.232708 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:54.221024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:40967)
0812 16:45:54.221555 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:54.221572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:54.221697 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:54.222094 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:54.222098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:54.222648 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:54.222815 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:54.222825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.224102 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.224106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:54.225126 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:54.225140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.225911 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.225915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:54.226041 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:54.226648 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:54.226670 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:54.232349 (+ 5679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":261,"threads_started":1}
W20260812 16:45:54.232882 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:54.221132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:40967 (local address 127.28.243.62:42539)
0812 16:45:54.221383 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:54.221389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:54.221685 (+ 296us) server_negotiation.cc:407] Connection header received
0812 16:45:54.221842 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:54.221846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:54.221915 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:54.222008 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:54.223028 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.223921 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.225312 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.225773 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.225878 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:54.232616 (+ 6738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:40967: BlockingRecv error: recv got EOF from 127.28.243.2:40967 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 16:45:54 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:54 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:54.739658 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:54.723159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:42941)
0812 16:45:54.723710 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:54.723727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:54.723868 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:54.724262 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:54.724266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:54.724699 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:54.724860 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:54.724869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.725972 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.725976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:54.726993 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:54.727007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.728401 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.728408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:54.728567 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:54.734626 (+ 6059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:54.734662 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:54.739395 (+ 4733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
W20260812 16:45:54.739971 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:54.723195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:42941 (local address 127.28.243.62:42539)
0812 16:45:54.723419 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:54.723424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:54.723804 (+ 380us) server_negotiation.cc:407] Connection header received
0812 16:45:54.724055 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:54.724059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:54.724137 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:54.724227 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:54.725038 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.725767 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.727317 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.728007 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.728187 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:54.739701 (+ 11514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:42941: BlockingRecv error: recv got EOF from 127.28.243.3:42941 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 16:45:54 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:54 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:54.863799 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:54.853164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:54885)
0812 16:45:54.853797 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:54.853817 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:54.853989 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:54.854338 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:54.854342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:54.854772 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:54.855043 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:54.855054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.856299 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.856304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:54.857724 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:54.857748 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.858809 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.858817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:54.859027 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:54.859739 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:54.859774 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:54.863489 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":326,"threads_started":1}
W20260812 16:45:54.864203 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:54.853338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:54885 (local address 127.28.243.62:42539)
0812 16:45:54.853629 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:54.853636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:54.854027 (+ 391us) server_negotiation.cc:407] Connection header received
0812 16:45:54.854103 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:54.854107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:54.854194 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:54.854293 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:54.855299 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.856098 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.857943 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:54.858484 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:54.858607 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:54.863872 (+ 5265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:54885: BlockingRecv error: recv got EOF from 127.28.243.1:54885 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 16:45:55 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:55 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:55.245313 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:55.234375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:49395)
0812 16:45:55.235133 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:55.235153 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:55.235311 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:55.235756 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:55.235760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:55.236169 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:55.236374 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:55.236383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.237729 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.237736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:55.239019 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:55.239041 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.240155 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.240163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:55.240343 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:55.241023 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:55.241044 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:55.245036 (+ 3992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":293,"threads_started":1}
W20260812 16:45:55.245530 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:55.234469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:49395 (local address 127.28.243.62:42539)
0812 16:45:55.234734 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:55.234742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:55.235327 (+ 585us) server_negotiation.cc:407] Connection header received
0812 16:45:55.235442 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:55.235447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:55.235545 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:55.235666 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:55.236592 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.237461 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.239274 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.239833 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.240013 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:55.245283 (+ 5270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:49395: BlockingRecv error: recv got EOF from 127.28.243.2:49395 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 12 16:45:55 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:55 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:55.751540 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:55.741042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:51053)
0812 16:45:55.741625 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:55.741643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:55.741760 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:55.742130 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:55.742135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:55.742564 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:55.742732 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:55.742740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.743986 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.743991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:55.745023 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:55.745039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.745884 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.745892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:55.746048 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:55.746982 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:55.747003 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:55.751172 (+ 4169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":316,"threads_started":1}
W20260812 16:45:55.751940 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:55.741259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:51053 (local address 127.28.243.62:42539)
0812 16:45:55.741473 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:55.741480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:55.741733 (+ 253us) server_negotiation.cc:407] Connection header received
0812 16:45:55.741875 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:55.741880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:55.741952 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:55.742084 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:55.742904 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.743803 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.745197 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.745660 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.745777 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:55.751608 (+ 5831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:51053: BlockingRecv error: recv got EOF from 127.28.243.3:51053 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 12 16:45:55 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:55 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:55.876282 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:55.865279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:54781)
0812 16:45:55.865810 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:55.865855 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:55.866012 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:55.866336 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:55.866339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:55.866785 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:55.867023 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:55.867035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.868483 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.868487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:55.869686 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:55.869702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.871221 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.871230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:55.871592 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:55.872340 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:55.872367 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:55.876030 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":295,"threads_started":1}
W20260812 16:45:55.876770 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:55.865447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:54781 (local address 127.28.243.62:42539)
0812 16:45:55.865648 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:55.865653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:55.865980 (+ 327us) server_negotiation.cc:407] Connection header received
0812 16:45:55.866131 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:55.866136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:55.866207 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:55.866301 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:55.867318 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.868244 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.869994 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:55.870836 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:55.871172 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:55.876448 (+ 5276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:54781: BlockingRecv error: recv got EOF from 127.28.243.1:54781 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 16:45:56 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:56 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:56.258605 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:56.246221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:58645)
0812 16:45:56.246731 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:56.246748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:56.246864 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:56.247448 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:56.247451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:56.247932 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:56.248152 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:56.248162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.249479 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.249483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:56.250464 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:56.250477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.251377 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.251382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:56.251529 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:56.252269 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:56.252405 (+ 136us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:56.258187 (+ 5782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":290,"threads_started":1}
W20260812 16:45:56.258797 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:56.246377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:58645 (local address 127.28.243.62:42539)
0812 16:45:56.246544 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:56.246549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:56.246818 (+ 269us) server_negotiation.cc:407] Connection header received
0812 16:45:56.247161 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:56.247167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:56.247273 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:56.247417 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:56.248349 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.249301 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.250668 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.251187 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.251306 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:56.258453 (+ 7147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:58645: BlockingRecv error: recv got EOF from 127.28.243.2:58645 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 16:45:56 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:56 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:56.764098 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:56.752831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:40433)
0812 16:45:56.753775 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:56.753819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:56.753994 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:56.754476 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:56.754481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:56.755109 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:56.755357 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:56.755369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.756633 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.756639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:56.758091 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:56.758125 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.759189 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.759197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:56.759407 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:56.760082 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:56.760104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:56.763774 (+ 3670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":677,"threads_started":1}
W20260812 16:45:56.764297 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:56.752878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:40433 (local address 127.28.243.62:42539)
0812 16:45:56.753237 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:56.753247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:56.754020 (+ 773us) server_negotiation.cc:407] Connection header received
0812 16:45:56.754106 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:56.754110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:56.754192 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:56.754304 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:56.755549 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.756420 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.758402 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.758961 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.759121 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:56.764044 (+ 4923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:40433: BlockingRecv error: recv got EOF from 127.28.243.3:40433 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 16:45:56 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:56 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:56.887755 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:56.877437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:50431)
0812 16:45:56.877999 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:56.878021 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:56.878179 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:56.878494 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:56.878497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:56.878907 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:56.879138 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:56.879147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.880213 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.880218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:56.881231 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:56.881246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.882194 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.882201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:56.882427 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:56.883325 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:56.883349 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:56.887405 (+ 4056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":272,"threads_started":1}
W20260812 16:45:56.887885 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:56.877571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:50431 (local address 127.28.243.62:42539)
0812 16:45:56.877868 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:56.877874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:56.878117 (+ 243us) server_negotiation.cc:407] Connection header received
0812 16:45:56.878302 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:56.878306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:56.878377 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:56.878465 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:56.879274 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.880066 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.881417 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:56.881885 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:56.882015 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:56.887631 (+ 5616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:50431: BlockingRecv error: recv got EOF from 127.28.243.1:50431 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 12 16:45:57 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:57 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:57.271402 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:57.259950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:34213)
0812 16:45:57.260554 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:57.260570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:57.260726 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:57.261064 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:57.261068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:57.261543 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:57.261787 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:57.261810 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.263001 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.263005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:57.264108 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:57.264124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.265158 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.265168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:57.265447 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:57.266359 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:57.266382 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:57.270793 (+ 4411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
W20260812 16:45:57.271581 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:57.260014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:34213 (local address 127.28.243.62:42539)
0812 16:45:57.260278 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:57.260285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:57.260677 (+ 392us) server_negotiation.cc:407] Connection header received
0812 16:45:57.260849 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:57.260853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:57.260917 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:57.261006 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:57.262023 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.262810 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.264270 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.264864 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.265013 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:57.271271 (+ 6258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:34213: BlockingRecv error: recv got EOF from 127.28.243.2:34213 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 12 16:45:57 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:57 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:57.779130 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:57.765321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:51483)
0812 16:45:57.765973 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:57.765994 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:57.766145 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:57.766534 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:57.766537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:57.767011 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:57.767212 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:57.767222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.768434 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.768441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:57.769998 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:57.770016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.771405 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.771412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:57.771622 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:57.772665 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:57.772703 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:57.778723 (+ 6020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":316,"threads_started":1}
W20260812 16:45:57.779362 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:57.765418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:51483 (local address 127.28.243.62:42539)
0812 16:45:57.765606 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:57.765613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:57.766084 (+ 471us) server_negotiation.cc:407] Connection header received
0812 16:45:57.766312 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:57.766316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:57.766399 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:57.766482 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:57.767382 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.768246 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.770213 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.771120 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.771245 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:57.779070 (+ 7825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:51483: BlockingRecv error: recv got EOF from 127.28.243.3:51483 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 16:45:57 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:57 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:57.900146 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:57.888879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:56597)
0812 16:45:57.889433 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:57.889450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:57.889776 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:57.890296 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:57.890302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:57.890875 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:57.891133 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:57.891144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.892329 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.892335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:57.893538 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:57.893553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.894437 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.894443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:57.894623 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:57.895537 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:57.895560 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:57.899853 (+ 4293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":339,"threads_started":1}
W20260812 16:45:57.900329 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:57.889031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:56597 (local address 127.28.243.62:42539)
0812 16:45:57.889256 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:57.889262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:57.889544 (+ 282us) server_negotiation.cc:407] Connection header received
0812 16:45:57.889977 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:57.889981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:57.890056 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:57.890160 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:57.891337 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.892089 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.893737 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:57.894195 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:57.894315 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:57.900095 (+ 5780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:56597: BlockingRecv error: recv got EOF from 127.28.243.1:56597 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 16:45:58 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:58 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:58.284297 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:58.273264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:34859)
0812 16:45:58.273957 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:58.273976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:58.274099 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:58.274662 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:58.274666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:58.275378 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:58.275584 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:58.275597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.277114 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.277120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:58.278143 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:58.278159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.279038 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.279044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:58.279202 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:58.280017 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:58.280038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:58.284012 (+ 3974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":298,"threads_started":1}
W20260812 16:45:58.284665 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:58.273476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:34859 (local address 127.28.243.62:42539)
0812 16:45:58.274016 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:58.274039 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:58.274062 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:45:58.274297 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:58.274330 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:58.274459 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:58.274625 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:58.275802 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.276838 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.278303 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.278796 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.278906 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:58.284368 (+ 5462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:34859: BlockingRecv error: recv got EOF from 127.28.243.2:34859 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Aug 12 16:45:58 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:58 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.3@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:58.790532 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:58.780655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.3:51043)
0812 16:45:58.781211 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:58.781229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:58.781368 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:58.781737 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:58.781742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:58.782211 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:58.782372 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:58.782381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.783445 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.783449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:58.784377 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:58.784389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.785149 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.785155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:58.785324 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:58.785969 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:58.785990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:58.789786 (+ 3796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
W20260812 16:45:58.790764 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:58.780685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:51043 (local address 127.28.243.62:42539)
0812 16:45:58.780865 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:58.780873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:58.781319 (+ 446us) server_negotiation.cc:407] Connection header received
0812 16:45:58.781495 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:58.781500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:58.781578 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:58.781688 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:58.782531 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.783302 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.784540 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.784986 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.785104 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:58.790497 (+ 5393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.3:51043: BlockingRecv error: recv got EOF from 127.28.243.3:51043 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 16:45:58 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:58 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.1@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:58.912257 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:58.901714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.1:43647)
0812 16:45:58.902239 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:58.902256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:58.902386 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:58.902781 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:58.902784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:58.903211 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:58.903411 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:58.903421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.904532 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.904536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:58.905555 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:58.905569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.906395 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.906401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:58.906555 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:58.907259 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:58.907285 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:58.911556 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
W20260812 16:45:58.912357 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:58.901801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:43647 (local address 127.28.243.62:42539)
0812 16:45:58.902120 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:58.902126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:58.902385 (+ 259us) server_negotiation.cc:407] Connection header received
0812 16:45:58.902491 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:58.902495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:58.902603 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:58.902691 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:58.903604 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.904343 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.905716 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:58.906187 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:58.906333 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:58.912045 (+ 5712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.1:43647: BlockingRecv error: recv got EOF from 127.28.243.1:43647 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Aug 12 16:45:59 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:45:59 dist-test-slave-1vmg krb5kdc[11472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.2@KRBTEST.COM for oryx/127.28.243.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:45:59.302042 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:59.286126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42539 (local address 127.28.243.2:35145)
0812 16:45:59.286872 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 16:45:59.286893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:45:59.287110 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:45:59.287715 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:45:59.287721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:45:59.288165 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:45:59.288423 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:59.288436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.290018 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.290024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:59.291507 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:45:59.291528 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.292621 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.292626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:45:59.292800 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:59.297252 (+ 4452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:45:59.297283 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:45:59.301734 (+ 4451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.62:42539: Server oryx/127.28.243.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":406,"threads_started":1}
W20260812 16:45:59.302295 13101 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:45:59.286252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:35145 (local address 127.28.243.62:42539)
0812 16:45:59.286443 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:59.286448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:59.287078 (+ 630us) server_negotiation.cc:407] Connection header received
0812 16:45:59.287312 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:59.287317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:59.287396 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:59.287533 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:59.288698 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.289755 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.291715 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.292396 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.292569 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:59.302039 (+ 9470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.2:35145: BlockingRecv error: recv got EOF from 127.28.243.2:35145 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:45:59.504293 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12421
I20260812 16:45:59.522528 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:59.505661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.28.243.1:32881)
0812 16:45:59.506346 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:59.506354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:59.506382 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:45:59.506452 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:59.506459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:59.506705 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:59.506947 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:59.508578 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.510096 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.512837 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.513852 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.514074 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:59.517495 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:59.517513 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:59.517534 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:59.517597 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:59.521696 (+ 4099us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 16:45:59.522184 (+ 488us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34692: Request ticket server kudu/127.28.243.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":374,"threads_started":1}
W20260812 16:45:59.522861 13181 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34692: Request ticket server kudu/127.28.243.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 16:45:59.522886 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12421: Not authorized: Client connection negotiation failed: client connection to 127.28.243.1:32881: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:45:59.523134 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 12421
I20260812 16:45:59.557276 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12680
W20260812 16:45:59.577492 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12680: Not authorized: Client connection negotiation failed: client connection to 127.28.243.2:43111: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:45:59.577560 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:59.559822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.28.243.2:43111)
0812 16:45:59.560767 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:59.560775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:59.560810 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:45:59.560884 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:59.560891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:59.561147 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:59.561377 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:59.562957 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.564176 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.565587 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.566241 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.566369 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:59.569298 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:59.569310 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:59.569327 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:59.569385 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:59.574695 (+ 5310us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 16:45:59.576067 (+ 1372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56688: Request ticket server kudu/127.28.243.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":420,"threads_started":1}
W20260812 16:45:59.577709 13182 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56688: Request ticket server kudu/127.28.243.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:45:59.577852 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 12680
I20260812 16:45:59.601388 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12901
I20260812 16:45:59.618256 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:45:59.602894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.28.243.3:40765)
0812 16:45:59.603732 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0812 16:45:59.603743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:45:59.603782 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:45:59.603882 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:45:59.603893 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:45:59.604289 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:45:59.604569 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:45:59.606058 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.607355 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.609171 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:45:59.610157 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:45:59.610312 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:45:59.614019 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:45:59.614031 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:45:59.614051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:45:59.614128 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:45:59.617504 (+ 3376us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 16:45:59.617938 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43090: Request ticket server kudu/127.28.243.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":379,"threads_started":1}
W20260812 16:45:59.618387 13184 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43090: Request ticket server kudu/127.28.243.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 16:45:59.618466 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12901: Not authorized: Client connection negotiation failed: client connection to 127.28.243.3:40765: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:45:59.618700 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 12901
I20260812 16:45:59.648032 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 12316
I20260812 16:45:59.649636 12411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:45:59.789687 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 12316
2026-08-12T16:45:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22744 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T16:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:45:59Z Disabled control of system clock
I20260812 16:45:59.911172 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36681
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:36681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 16:46:00.345917 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:00.346288 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:00.346380 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:00.357169 13194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:46:00.357372 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:00.357416 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:00.357512 13194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:46:00.357583 13194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:46:00.368939 13194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:36681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:00.370904 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:00.373746 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:00.388547 13204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:00.388906 13201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:00.388932 13208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:00.390015 13194 server_base.cc:1034] running on GCE node
I20260812 16:46:00.391572 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:00.393694 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:00.395045 13194 hybrid_clock.cc:648] HybridClock initialized: now 1786553160394966 us; error 69 us; skew 500 ppm
I20260812 16:46:00.395687 13194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:00.401969 13194 webserver.cc:460] Webserver started at http://127.28.243.62:38809/ using document root <none> and password file <none>
I20260812 16:46:00.402890 13194 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:00.403055 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:00.403437 13194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:00.406847 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "625bd02792e347d6b2bebadccfe9c82b"
format_stamp: "Formatted at 2026-08-12 16:46:00 on dist-test-slave-1vmg"
server_key: "c07749153126e545b583db5b72053a3d"
server_key_iv: "3fac9678b297faf13b2cd10c55c51100"
server_key_version: "encryptionkey@0"
I20260812 16:46:00.407852 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "625bd02792e347d6b2bebadccfe9c82b"
format_stamp: "Formatted at 2026-08-12 16:46:00 on dist-test-slave-1vmg"
server_key: "c07749153126e545b583db5b72053a3d"
server_key_iv: "3fac9678b297faf13b2cd10c55c51100"
server_key_version: "encryptionkey@0"
I20260812 16:46:00.415540 13194 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260812 16:46:00.420657 13221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:00.422429 13194 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:46:00.422660 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "625bd02792e347d6b2bebadccfe9c82b"
format_stamp: "Formatted at 2026-08-12 16:46:00 on dist-test-slave-1vmg"
server_key: "c07749153126e545b583db5b72053a3d"
server_key_iv: "3fac9678b297faf13b2cd10c55c51100"
server_key_version: "encryptionkey@0"
I20260812 16:46:00.422972 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:00.471293 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:00.473755 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:00.474212 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:00.498560 13194 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:36681
I20260812 16:46:00.498585 13310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:36681 every 8 connection(s)
I20260812 16:46:00.500524 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:46:00.503448 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 13194
I20260812 16:46:00.503671 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:46:00.504560 29644 external_mini_cluster.cc:1428] Setting key ea5d633f1b0ccf6f9fa9f171582f1017
I20260812 16:46:00.505718 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:46:00.522967 13311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b: Bootstrap starting.
I20260812 16:46:00.525211 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:00.507219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.28.243.62:36681)
0812 16:46:00.508328 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:00.508340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:00.508396 (+ 56us) server_negotiation.cc:407] Connection header received
0812 16:46:00.512006 (+ 3610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:00.512054 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:00.512908 (+ 854us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:00.513504 (+ 596us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:00.514360 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:00.516933 (+ 2573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:00.519682 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:00.520597 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:00.520839 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:00.522357 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:00.522367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:00.522386 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:00.523088 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:00.523240 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:00.523268 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:00.523402 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:00.523689 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 16:46:00.524073 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":393,"threads_started":1}
I20260812 16:46:00.526767 13311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b: Neither blocks nor log segments found. Creating new log.
I20260812 16:46:00.528057 13311 log.cc:824] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:46:00.533290 13311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b: No bootstrap required, opened a new log
I20260812 16:46:00.544399 13311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625bd02792e347d6b2bebadccfe9c82b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36681 } }
I20260812 16:46:00.544870 13311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:46:00.544966 13311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625bd02792e347d6b2bebadccfe9c82b, State: Initialized, Role: FOLLOWER
I20260812 16:46:00.545699 13311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625bd02792e347d6b2bebadccfe9c82b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36681 } }
I20260812 16:46:00.546036 13311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:46:00.546188 13311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:46:00.546461 13311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:46:00.549599 13311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625bd02792e347d6b2bebadccfe9c82b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36681 } }
I20260812 16:46:00.550585 13311 leader_election.cc:302] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625bd02792e347d6b2bebadccfe9c82b; no voters:
I20260812 16:46:00.551496 13311 leader_election.cc:288] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:46:00.551740 13316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:46:00.553704 13316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [term 1 LEADER]: Becoming Leader. State: Replica: 625bd02792e347d6b2bebadccfe9c82b, State: Running, Role: LEADER
I20260812 16:46:00.554471 13316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625bd02792e347d6b2bebadccfe9c82b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36681 } }
I20260812 16:46:00.554831 13311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:46:00.558102 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625bd02792e347d6b2bebadccfe9c82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625bd02792e347d6b2bebadccfe9c82b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36681 } } }
I20260812 16:46:00.558789 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [sys.catalog]: This master's current role is: LEADER
I20260812 16:46:00.558060 13318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625bd02792e347d6b2bebadccfe9c82b. Latest consensus state: current_term: 1 leader_uuid: "625bd02792e347d6b2bebadccfe9c82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625bd02792e347d6b2bebadccfe9c82b" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36681 } } }
I20260812 16:46:00.559515 13318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b [sys.catalog]: This master's current role is: LEADER
I20260812 16:46:00.566170 13325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:46:00.575425 13325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:46:00.586829 13325 catalog_manager.cc:1347] Generated new cluster ID: c11562f72a5645db875f38c9718cfcea
I20260812 16:46:00.587082 13325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:46:00.603686 13325 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:46:00.605262 13325 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:46:00.615314 13325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625bd02792e347d6b2bebadccfe9c82b: Generated new TSK 0
I20260812 16:46:00.616604 13325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:46:00.627888 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36681
--builtin_ntp_servers=127.28.243.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 16:46:01.106252 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:01.106581 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:01.106685 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:01.116181 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:01.116532 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:01.116915 13343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:46:01.127897 13343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:01.129302 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:01.132301 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:01.148937 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:01.148955 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:01.149221 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:01.150225 13343 server_base.cc:1034] running on GCE node
I20260812 16:46:01.151405 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:01.152808 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:01.154091 13343 hybrid_clock.cc:648] HybridClock initialized: now 1786553161154019 us; error 70 us; skew 500 ppm
I20260812 16:46:01.154505 13343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:01.158703 13343 webserver.cc:460] Webserver started at http://127.28.243.1:37151/ using document root <none> and password file <none>
I20260812 16:46:01.159585 13343 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:01.159708 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:01.160106 13343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:01.163437 13343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "ac8477fbd7814226b633f1fe2c59cb00"
format_stamp: "Formatted at 2026-08-12 16:46:01 on dist-test-slave-1vmg"
server_key: "508bc5bcf803a39eedca9ebebe05bc70"
server_key_iv: "a3faf3cc322f8779ef6ec65eda9a33b7"
server_key_version: "encryptionkey@0"
I20260812 16:46:01.164374 13343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "ac8477fbd7814226b633f1fe2c59cb00"
format_stamp: "Formatted at 2026-08-12 16:46:01 on dist-test-slave-1vmg"
server_key: "508bc5bcf803a39eedca9ebebe05bc70"
server_key_iv: "a3faf3cc322f8779ef6ec65eda9a33b7"
server_key_version: "encryptionkey@0"
I20260812 16:46:01.170229 13343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 16:46:01.175243 13371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:01.177599 13343 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 16:46:01.178570 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "ac8477fbd7814226b633f1fe2c59cb00"
format_stamp: "Formatted at 2026-08-12 16:46:01 on dist-test-slave-1vmg"
server_key: "508bc5bcf803a39eedca9ebebe05bc70"
server_key_iv: "a3faf3cc322f8779ef6ec65eda9a33b7"
server_key_version: "encryptionkey@0"
I20260812 16:46:01.178845 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:01.216570 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:01.218151 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:01.218549 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:01.219990 13343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:01.222162 13343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:01.222280 13343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:01.222396 13343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:01.222457 13343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:01.263677 13343 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:42361
I20260812 16:46:01.263731 13545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:42361 every 8 connection(s)
I20260812 16:46:01.265158 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:46:01.272308 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 13343
I20260812 16:46:01.272763 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:46:01.273856 29644 external_mini_cluster.cc:1428] Setting key 7aa1ef96d22989b4c7e0b494942f965a
I20260812 16:46:01.278883 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36681
--builtin_ntp_servers=127.28.243.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 16:46:01.286284 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:01.269176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:36619 (local address 127.28.243.62:36681)
0812 16:46:01.270112 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:01.270120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:01.273093 (+ 2973us) server_negotiation.cc:407] Connection header received
0812 16:46:01.276107 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:01.276114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:01.276209 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:01.276371 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:01.279029 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.279930 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.282570 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.283139 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.283263 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:01.285220 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:01.285227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:01.285231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:01.285262 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:01.285342 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:01.285359 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:01.285470 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:01.285834 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 16:46:01.285954 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":414,"threads_started":1}
I20260812 16:46:01.287734 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:01.270483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36681 (local address 127.28.243.1:36619)
0812 16:46:01.272746 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:01.272832 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:01.275561 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:01.276810 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:01.276823 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:01.277153 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:01.278648 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:01.278702 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.280156 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.280164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:01.282236 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:01.282268 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.283447 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.283455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:01.283813 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:01.284678 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:46:01.284681 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:01.284749 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:46:01.284875 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:01.285502 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:01.285513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:01.285522 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:46:01.285759 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 16:46:01.286229 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":765,"threads_started":1}
I20260812 16:46:01.289693 13546 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36681
I20260812 16:46:01.290145 13546 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:01.291111 13546 heartbeater.cc:511] Master 127.28.243.62:36681 requested a full tablet report, sending...
I20260812 16:46:01.293689 13244 ts_manager.cc:194] Registered new tserver with Master: ac8477fbd7814226b633f1fe2c59cb00 (127.28.243.1:42361)
I20260812 16:46:01.295924 13244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.1:36619
W20260812 16:46:01.695025 13553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:01.695482 13553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:01.695603 13553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:01.703552 13553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:01.703692 13553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:01.703833 13553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:46:01.713601 13553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:01.715183 13553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:01.717284 13553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:01.734037 13562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:01.734020 13568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:01.733958 13563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:01.734822 13553 server_base.cc:1034] running on GCE node
I20260812 16:46:01.735534 13553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:01.736728 13553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:01.738013 13553 hybrid_clock.cc:648] HybridClock initialized: now 1786553161737931 us; error 76 us; skew 500 ppm
I20260812 16:46:01.738583 13553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:01.741441 13553 webserver.cc:460] Webserver started at http://127.28.243.2:36711/ using document root <none> and password file <none>
I20260812 16:46:01.742274 13553 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:01.742381 13553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:01.742725 13553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:01.746039 13553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "b64d4f58831f4a62bf83a8526f8b25af"
format_stamp: "Formatted at 2026-08-12 16:46:01 on dist-test-slave-1vmg"
server_key: "f951b013da382033ce38184e8f3cd9b7"
server_key_iv: "3a260b2d8c2c1363f08076bf817ef2eb"
server_key_version: "encryptionkey@0"
I20260812 16:46:01.747061 13553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "b64d4f58831f4a62bf83a8526f8b25af"
format_stamp: "Formatted at 2026-08-12 16:46:01 on dist-test-slave-1vmg"
server_key: "f951b013da382033ce38184e8f3cd9b7"
server_key_iv: "3a260b2d8c2c1363f08076bf817ef2eb"
server_key_version: "encryptionkey@0"
I20260812 16:46:01.752913 13553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:46:01.757216 13577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:01.758733 13553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:46:01.758908 13553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "b64d4f58831f4a62bf83a8526f8b25af"
format_stamp: "Formatted at 2026-08-12 16:46:01 on dist-test-slave-1vmg"
server_key: "f951b013da382033ce38184e8f3cd9b7"
server_key_iv: "3a260b2d8c2c1363f08076bf817ef2eb"
server_key_version: "encryptionkey@0"
I20260812 16:46:01.759202 13553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:01.780462 13553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:01.781461 13553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:01.781780 13553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:01.782819 13553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:01.784482 13553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:01.784583 13553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:01.784691 13553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:01.784782 13553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:01.821398 13553 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:33193
I20260812 16:46:01.821485 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:33193 every 8 connection(s)
I20260812 16:46:01.822863 13553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:46:01.829305 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 13553
I20260812 16:46:01.829489 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:46:01.829944 29644 external_mini_cluster.cc:1428] Setting key d37b9a39f0120a19e4123264a516f39d
I20260812 16:46:01.833472 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36681
--builtin_ntp_servers=127.28.243.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 16:46:01.840561 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:01.825936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:47297 (local address 127.28.243.62:36681)
0812 16:46:01.826576 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:01.826583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:01.828317 (+ 1734us) server_negotiation.cc:407] Connection header received
0812 16:46:01.830624 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:01.830632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:01.830756 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:01.831178 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:01.832763 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.833579 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.836332 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.836900 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.837069 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:01.839293 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:01.839306 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:01.839310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:01.839346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:01.839422 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:01.839440 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:01.839557 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:01.840022 (+ 465us) server_negotiation.cc:299] Negotiation successful
0812 16:46:01.840164 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":282,"threads_started":1}
I20260812 16:46:01.841902 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:01.826463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36681 (local address 127.28.243.2:47297)
0812 16:46:01.828031 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:01.828102 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:01.830208 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:01.831453 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:01.831465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:01.831596 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:01.832447 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:01.832482 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.833863 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.833869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:01.836021 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:01.836043 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:01.837257 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:01.837265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:01.837621 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:01.838681 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:46:01.838685 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:01.838750 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:46:01.838887 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:01.839591 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:01.839602 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:01.839619 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 16:46:01.839886 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 16:46:01.840399 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":445,"threads_started":1}
I20260812 16:46:01.844215 13759 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36681
I20260812 16:46:01.844797 13759 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:01.846143 13759 heartbeater.cc:511] Master 127.28.243.62:36681 requested a full tablet report, sending...
I20260812 16:46:01.848476 13244 ts_manager.cc:194] Registered new tserver with Master: b64d4f58831f4a62bf83a8526f8b25af (127.28.243.2:33193)
I20260812 16:46:01.849620 13244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.2:47297
W20260812 16:46:02.232781 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:02.233160 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:02.233256 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:02.241065 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:02.241206 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:02.241320 13769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:46:02.251885 13769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:02.253361 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:02.255455 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:02.270200 13777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:02.270171 13776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:02.270613 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:02.271296 13769 server_base.cc:1034] running on GCE node
I20260812 16:46:02.272015 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:02.273191 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:02.274437 13769 hybrid_clock.cc:648] HybridClock initialized: now 1786553162274372 us; error 59 us; skew 500 ppm
I20260812 16:46:02.274796 13769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:02.277244 13769 webserver.cc:460] Webserver started at http://127.28.243.3:46119/ using document root <none> and password file <none>
I20260812 16:46:02.278014 13769 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:02.278126 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:02.278460 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:02.281774 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "b316c56b941b41269773c0f3661e5217"
format_stamp: "Formatted at 2026-08-12 16:46:02 on dist-test-slave-1vmg"
server_key: "7d4a28840d6c8e6a3c8837d429136c93"
server_key_iv: "1d5eca36ba751e3921e9b6e81d2cd7f4"
server_key_version: "encryptionkey@0"
I20260812 16:46:02.282774 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "b316c56b941b41269773c0f3661e5217"
format_stamp: "Formatted at 2026-08-12 16:46:02 on dist-test-slave-1vmg"
server_key: "7d4a28840d6c8e6a3c8837d429136c93"
server_key_iv: "1d5eca36ba751e3921e9b6e81d2cd7f4"
server_key_version: "encryptionkey@0"
I20260812 16:46:02.289597 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:46:02.294533 13786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:02.296497 13769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:46:02.296736 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "b316c56b941b41269773c0f3661e5217"
format_stamp: "Formatted at 2026-08-12 16:46:02 on dist-test-slave-1vmg"
server_key: "7d4a28840d6c8e6a3c8837d429136c93"
server_key_iv: "1d5eca36ba751e3921e9b6e81d2cd7f4"
server_key_version: "encryptionkey@0"
I20260812 16:46:02.296983 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:02.300693 13546 heartbeater.cc:503] Master 127.28.243.62:36681 was elected leader, sending a full tablet report...
I20260812 16:46:02.349258 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:02.350725 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:02.351199 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:02.352639 13769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:02.354696 13769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:02.354825 13769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:02.354959 13769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:02.355039 13769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 16:46:02.392721 13769 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:33397
I20260812 16:46:02.392761 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:33397 every 8 connection(s)
I20260812 16:46:02.394608 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:46:02.399729 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 13769
I20260812 16:46:02.399945 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:46:02.400745 29644 external_mini_cluster.cc:1428] Setting key 576002ae2746a44016a21dfe033946b9
I20260812 16:46:02.409657 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:02.397669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:40929 (local address 127.28.243.62:36681)
0812 16:46:02.398237 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:02.398245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:02.399740 (+ 1495us) server_negotiation.cc:407] Connection header received
0812 16:46:02.401850 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:02.401855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:02.401956 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:02.402082 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:02.403737 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.404772 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.406491 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.406980 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.407104 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:02.408713 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:02.408719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:02.408722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:02.408745 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:02.408789 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:02.408803 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:02.408885 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:02.409134 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 16:46:02.409252 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":252,"threads_started":1}
I20260812 16:46:02.410951 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:02.398297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36681 (local address 127.28.243.3:40929)
0812 16:46:02.399520 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:02.399570 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:02.401494 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:02.402413 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:02.402427 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:02.402565 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:02.403521 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:02.403552 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.404933 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.404938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:02.406325 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:02.406339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.407238 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.407247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:02.407497 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:02.408307 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:46:02.408311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:02.408375 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:46:02.408486 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:02.408899 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:02.408912 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:02.408921 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:46:02.409125 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 16:46:02.409598 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":366,"threads_started":1}
I20260812 16:46:02.420797 13991 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36681
I20260812 16:46:02.421350 13991 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:02.422535 13991 heartbeater.cc:511] Master 127.28.243.62:36681 requested a full tablet report, sending...
I20260812 16:46:02.424873 13244 ts_manager.cc:194] Registered new tserver with Master: b316c56b941b41269773c0f3661e5217 (127.28.243.3:33397)
I20260812 16:46:02.425796 13244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.3:40929
I20260812 16:46:02.431038 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 16:46:02.440044 13996 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:46:02.437976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.28.243.62:36681)
0812 16:46:02.438225 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:02.438232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:02.438634 (+ 402us) server_negotiation.cc:407] Connection header received
0812 16:46:02.438820 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:02.438827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:02.438957 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:02.439087 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:02.439671 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55482: BlockingRecv error: recv got EOF from 127.0.0.1:55482 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:46:02.440613 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 13343
I20260812 16:46:02.449290 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:02.441761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.28.243.1:42361)
0812 16:46:02.442385 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:02.442392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:02.442420 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:46:02.442484 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:02.442491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:02.442733 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:02.442892 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:02.443377 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.444437 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.445768 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.446384 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.446511 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:02.447725 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:02.447734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:02.447749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:02.448399 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:02.448452 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:02.448475 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:02.448601 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:02.448765 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 16:46:02.448921 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":306,"threads_started":1}
I20260812 16:46:02.450649 13540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:02.547429 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 13343
I20260812 16:46:02.581527 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 13553
I20260812 16:46:02.592396 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:02.582794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.28.243.2:33193)
0812 16:46:02.583409 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:02.583417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:02.583451 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:46:02.583533 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:02.583540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:02.583889 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:02.584092 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:02.584584 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.585618 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.587212 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.588369 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.588558 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:02.590075 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:02.590091 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:02.590118 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:02.591175 (+ 1057us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:02.591267 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:02.591298 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:02.591434 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:02.591751 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 16:46:02.592004 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":316,"threads_started":1}
I20260812 16:46:02.594329 13754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:02.681237 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 13553
I20260812 16:46:02.708408 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 13769
I20260812 16:46:02.719075 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:02.709567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.28.243.3:33397)
0812 16:46:02.710453 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:02.710463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:02.710514 (+ 51us) server_negotiation.cc:407] Connection header received
0812 16:46:02.710626 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:02.710633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:02.711017 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:02.711261 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:02.712156 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.713650 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.715357 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:02.716035 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:02.716175 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:02.717396 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:02.717406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:02.717424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:02.718101 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:02.718158 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:02.718181 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:02.718318 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:02.718471 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 16:46:02.718646 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":312,"threads_started":1}
I20260812 16:46:02.720618 13982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:02.808219 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 13769
I20260812 16:46:02.837462 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 13194
I20260812 16:46:02.838838 13303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:02.955950 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 13194
2026-08-12T16:46:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3157 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T16:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:46:03Z Disabled control of system clock
I20260812 16:46:03.062104 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36453
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 16:46:03.462095 14021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:03.462447 14021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:03.462545 14021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:03.471180 14021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:46:03.471321 14021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:03.471360 14021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:03.471421 14021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:46:03.471472 14021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:46:03.481931 14021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:36453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:03.483415 14021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:03.485921 14021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:03.502575 14030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:03.502621 14033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:03.503068 14029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:03.508941 14021 server_base.cc:1034] running on GCE node
I20260812 16:46:03.509914 14021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:03.511679 14021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:03.512965 14021 hybrid_clock.cc:648] HybridClock initialized: now 1786553163512892 us; error 77 us; skew 500 ppm
I20260812 16:46:03.513430 14021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:03.516355 14021 webserver.cc:460] Webserver started at http://127.28.243.62:39755/ using document root <none> and password file <none>
I20260812 16:46:03.517208 14021 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:03.517335 14021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:03.517700 14021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:03.521360 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "5c3f6143022a4ad8a1b049e4e81eabe2"
format_stamp: "Formatted at 2026-08-12 16:46:03 on dist-test-slave-1vmg"
server_key: "2f3412aae5eebb6514dcc8891ce84906"
server_key_iv: "030daa726a740367de7846b9a10de5b7"
server_key_version: "encryptionkey@0"
I20260812 16:46:03.522743 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "5c3f6143022a4ad8a1b049e4e81eabe2"
format_stamp: "Formatted at 2026-08-12 16:46:03 on dist-test-slave-1vmg"
server_key: "2f3412aae5eebb6514dcc8891ce84906"
server_key_iv: "030daa726a740367de7846b9a10de5b7"
server_key_version: "encryptionkey@0"
I20260812 16:46:03.529809 14021 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 16:46:03.534662 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:03.537191 14021 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 16:46:03.537438 14021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "5c3f6143022a4ad8a1b049e4e81eabe2"
format_stamp: "Formatted at 2026-08-12 16:46:03 on dist-test-slave-1vmg"
server_key: "2f3412aae5eebb6514dcc8891ce84906"
server_key_iv: "030daa726a740367de7846b9a10de5b7"
server_key_version: "encryptionkey@0"
I20260812 16:46:03.537720 14021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:03.544585 14021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:03.545528 14021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:03.545859 14021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:03.567550 14021 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:36453
I20260812 16:46:03.567574 14116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:36453 every 8 connection(s)
I20260812 16:46:03.569078 14021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:46:03.574740 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:46:03.577800 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 14021
I20260812 16:46:03.578028 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:46:03.578662 29644 external_mini_cluster.cc:1428] Setting key 051e3880cfc4914f3ef6e2a336c2632c
I20260812 16:46:03.590262 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2: Bootstrap starting.
I20260812 16:46:03.590863 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:03.581616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.28.243.62:36453)
0812 16:46:03.583337 (+ 1721us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:03.583376 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:03.583449 (+ 73us) server_negotiation.cc:407] Connection header received
0812 16:46:03.586107 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:03.586143 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:03.586714 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:03.587265 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:03.588462 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:03.588470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:03.588484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:03.588501 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:03.588574 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:03.588596 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:03.588831 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:03.589157 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 16:46:03.589612 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":560,"threads_started":1}
I20260812 16:46:03.595320 14120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2: Neither blocks nor log segments found. Creating new log.
I20260812 16:46:03.597052 14120 log.cc:824] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2: Log is configured to *not* fsync() on all Append() calls
I20260812 16:46:03.600651 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2: No bootstrap required, opened a new log
I20260812 16:46:03.608877 14120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36453 } }
I20260812 16:46:03.609323 14120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:46:03.609421 14120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3f6143022a4ad8a1b049e4e81eabe2, State: Initialized, Role: FOLLOWER
I20260812 16:46:03.610553 14120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36453 } }
I20260812 16:46:03.610882 14120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:46:03.611076 14120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:46:03.611266 14120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:46:03.614408 14120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36453 } }
I20260812 16:46:03.615410 14120 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3f6143022a4ad8a1b049e4e81eabe2; no voters:
I20260812 16:46:03.616262 14120 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:46:03.616477 14127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:46:03.617388 14127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3f6143022a4ad8a1b049e4e81eabe2, State: Running, Role: LEADER
I20260812 16:46:03.618070 14127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36453 } }
I20260812 16:46:03.619453 14120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:46:03.621194 14130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3f6143022a4ad8a1b049e4e81eabe2. Latest consensus state: current_term: 1 leader_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36453 } } }
I20260812 16:46:03.621218 14128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f6143022a4ad8a1b049e4e81eabe2" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36453 } } }
I20260812 16:46:03.621726 14130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [sys.catalog]: This master's current role is: LEADER
I20260812 16:46:03.621734 14128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2 [sys.catalog]: This master's current role is: LEADER
I20260812 16:46:03.624013 14137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:46:03.630084 14137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:46:03.641918 14137 catalog_manager.cc:1347] Generated new cluster ID: 98ff14e2acb04a738254f45ae7933250
I20260812 16:46:03.642072 14137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:46:03.675168 14137 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:46:03.676092 14137 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:46:03.688030 14137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3f6143022a4ad8a1b049e4e81eabe2: Generated new TSK 0
I20260812 16:46:03.689297 14137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:46:03.699460 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36453
--builtin_ntp_servers=127.28.243.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 16:46:04.157729 14162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:04.158187 14162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:04.158350 14162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:04.167320 14162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:04.167464 14162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:04.167634 14162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:46:04.178718 14162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:04.180259 14162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:04.182271 14162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:04.197991 14172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:04.198242 14170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:04.197992 14174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:04.198983 14162 server_base.cc:1034] running on GCE node
I20260812 16:46:04.199635 14162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:04.200735 14162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:04.202004 14162 hybrid_clock.cc:648] HybridClock initialized: now 1786553164201940 us; error 47 us; skew 500 ppm
I20260812 16:46:04.202368 14162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:04.205189 14162 webserver.cc:460] Webserver started at http://127.28.243.1:38451/ using document root <none> and password file <none>
I20260812 16:46:04.205941 14162 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:04.206045 14162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:04.206326 14162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:04.209882 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "8601b1000d2e441e8e493fdc151f3d41"
format_stamp: "Formatted at 2026-08-12 16:46:04 on dist-test-slave-1vmg"
server_key: "80d8c4a4dca3f7add7e76a3a1f7fefe8"
server_key_iv: "1dd242afef61a130d9d52da3c3673f23"
server_key_version: "encryptionkey@0"
I20260812 16:46:04.210732 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "8601b1000d2e441e8e493fdc151f3d41"
format_stamp: "Formatted at 2026-08-12 16:46:04 on dist-test-slave-1vmg"
server_key: "80d8c4a4dca3f7add7e76a3a1f7fefe8"
server_key_iv: "1dd242afef61a130d9d52da3c3673f23"
server_key_version: "encryptionkey@0"
I20260812 16:46:04.217099 14162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 16:46:04.223079 14184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:04.225633 14162 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:46:04.225893 14162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "8601b1000d2e441e8e493fdc151f3d41"
format_stamp: "Formatted at 2026-08-12 16:46:04 on dist-test-slave-1vmg"
server_key: "80d8c4a4dca3f7add7e76a3a1f7fefe8"
server_key_iv: "1dd242afef61a130d9d52da3c3673f23"
server_key_version: "encryptionkey@0"
I20260812 16:46:04.226179 14162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:04.233309 14162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:04.234324 14162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:04.234637 14162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:04.236052 14162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:04.237785 14162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:04.237890 14162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:04.237975 14162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:04.238026 14162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:04.276963 14162 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:46135
I20260812 16:46:04.277020 14370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:46135 every 8 connection(s)
I20260812 16:46:04.278380 14162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:46:04.280714 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 14162
I20260812 16:46:04.280959 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:46:04.281641 29644 external_mini_cluster.cc:1428] Setting key aaf2ee8ef689dd87fdcd40103555c5c2
I20260812 16:46:04.285179 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36453
--builtin_ntp_servers=127.28.243.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 16:46:04.289088 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:04.281610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:58391 (local address 127.28.243.62:36453)
0812 16:46:04.282129 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:04.282135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:04.286091 (+ 3956us) server_negotiation.cc:407] Connection header received
0812 16:46:04.286167 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:04.286173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:04.286241 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:04.286349 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:04.287888 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:04.287893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:04.287898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:04.287921 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:04.287978 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:04.287994 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:04.288130 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:04.288381 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 16:46:04.288452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":251,"threads_started":1}
I20260812 16:46:04.289979 14376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:04.282265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36453 (local address 127.28.243.1:58391)
0812 16:46:04.283450 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:04.283500 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:04.285682 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:04.286637 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:04.286647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:04.286771 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:04.287527 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:46:04.287530 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:04.287576 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:46:04.287678 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:04.288140 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:04.288152 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:04.288161 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:46:04.288461 (+ 300us) client_negotiation.cc:240] Negotiation successful
0812 16:46:04.288747 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":390,"threads_started":1}
I20260812 16:46:04.292618 14371 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36453
I20260812 16:46:04.293704 14371 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:04.295761 14371 heartbeater.cc:511] Master 127.28.243.62:36453 requested a full tablet report, sending...
I20260812 16:46:04.299906 14066 ts_manager.cc:194] Registered new tserver with Master: 8601b1000d2e441e8e493fdc151f3d41 (127.28.243.1:46135)
W20260812 16:46:04.685745 14377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:04.686074 14377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:04.686197 14377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:04.695333 14377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:04.695475 14377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:04.695602 14377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:46:04.707430 14377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:04.708976 14377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:04.711054 14377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:04.726074 14387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:04.725994 14388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:04.726121 14394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:04.726614 14377 server_base.cc:1034] running on GCE node
I20260812 16:46:04.728004 14377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:04.729700 14377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:04.731061 14377 hybrid_clock.cc:648] HybridClock initialized: now 1786553164731006 us; error 58 us; skew 500 ppm
I20260812 16:46:04.731551 14377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:04.734707 14377 webserver.cc:460] Webserver started at http://127.28.243.2:37853/ using document root <none> and password file <none>
I20260812 16:46:04.735641 14377 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:04.735767 14377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:04.736137 14377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:04.739663 14377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "e869c896d45641908604c05b44e0253f"
format_stamp: "Formatted at 2026-08-12 16:46:04 on dist-test-slave-1vmg"
server_key: "3e59f410395f023e47b3b9e5e08cbaeb"
server_key_iv: "ebbfa7fa121a769d1bcc83e92ae79e7b"
server_key_version: "encryptionkey@0"
I20260812 16:46:04.740651 14377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "e869c896d45641908604c05b44e0253f"
format_stamp: "Formatted at 2026-08-12 16:46:04 on dist-test-slave-1vmg"
server_key: "3e59f410395f023e47b3b9e5e08cbaeb"
server_key_iv: "ebbfa7fa121a769d1bcc83e92ae79e7b"
server_key_version: "encryptionkey@0"
I20260812 16:46:04.747622 14377 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 16:46:04.753453 14406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:04.755653 14377 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 16:46:04.755897 14377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "e869c896d45641908604c05b44e0253f"
format_stamp: "Formatted at 2026-08-12 16:46:04 on dist-test-slave-1vmg"
server_key: "3e59f410395f023e47b3b9e5e08cbaeb"
server_key_iv: "ebbfa7fa121a769d1bcc83e92ae79e7b"
server_key_version: "encryptionkey@0"
I20260812 16:46:04.756179 14377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:04.763317 14377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:04.764312 14377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:04.764604 14377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:04.765704 14377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:04.767426 14377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:04.767537 14377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:04.767661 14377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:04.767735 14377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:04.807153 14377 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:42619
I20260812 16:46:04.807235 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:42619 every 8 connection(s)
I20260812 16:46:04.808595 14377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:46:04.810000 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 14377
I20260812 16:46:04.810223 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:46:04.811160 29644 external_mini_cluster.cc:1428] Setting key 1473de3a137528146d9993cfcaa690c1
I20260812 16:46:04.815976 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36453
--builtin_ntp_servers=127.28.243.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 16:46:04.821617 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:04.811886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:54905 (local address 127.28.243.62:36453)
0812 16:46:04.812564 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:04.812572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:04.814297 (+ 1725us) server_negotiation.cc:407] Connection header received
0812 16:46:04.817719 (+ 3422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:04.817731 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:04.817821 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:04.817974 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:04.820638 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:04.820644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:04.820649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:04.820678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:04.820742 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:04.820757 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:04.820912 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:04.821207 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 16:46:04.821283 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":322,"threads_started":1}
I20260812 16:46:04.822747 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:04.812472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36453 (local address 127.28.243.2:54905)
0812 16:46:04.814010 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:04.814097 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:04.817080 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:04.818428 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:04.818444 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:04.818639 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:04.820000 (+ 1361us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:46:04.820005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:04.820099 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:46:04.820285 (+ 186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:04.820921 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:04.820933 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:04.820957 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:46:04.821316 (+ 359us) client_negotiation.cc:240] Negotiation successful
0812 16:46:04.821610 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":450,"threads_started":1}
I20260812 16:46:04.825131 14585 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36453
I20260812 16:46:04.825639 14585 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:04.826674 14585 heartbeater.cc:511] Master 127.28.243.62:36453 requested a full tablet report, sending...
I20260812 16:46:04.829044 14066 ts_manager.cc:194] Registered new tserver with Master: e869c896d45641908604c05b44e0253f (127.28.243.2:42619)
W20260812 16:46:05.209178 14590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:05.209542 14590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:05.209645 14590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:05.218053 14590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:05.218194 14590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:05.218313 14590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:46:05.228950 14590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:05.230331 14590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:05.232245 14590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:05.247125 14596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:05.247051 14598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:05.247097 14600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:05.247897 14590 server_base.cc:1034] running on GCE node
I20260812 16:46:05.248611 14590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:05.250161 14590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:05.251438 14590 hybrid_clock.cc:648] HybridClock initialized: now 1786553165251376 us; error 56 us; skew 500 ppm
I20260812 16:46:05.251829 14590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:46:05.254554 14590 webserver.cc:460] Webserver started at http://127.28.243.3:36905/ using document root <none> and password file <none>
I20260812 16:46:05.255403 14590 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:05.255527 14590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:05.255880 14590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:05.259790 14590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "167bfe20ec0446c3b2968faf0b11ef4a"
format_stamp: "Formatted at 2026-08-12 16:46:05 on dist-test-slave-1vmg"
server_key: "1ac6551642f0ad485e1899bdb6415963"
server_key_iv: "20f60b199b31f301e982cb34df01e646"
server_key_version: "encryptionkey@0"
I20260812 16:46:05.260790 14590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "167bfe20ec0446c3b2968faf0b11ef4a"
format_stamp: "Formatted at 2026-08-12 16:46:05 on dist-test-slave-1vmg"
server_key: "1ac6551642f0ad485e1899bdb6415963"
server_key_iv: "20f60b199b31f301e982cb34df01e646"
server_key_version: "encryptionkey@0"
I20260812 16:46:05.267683 14590 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:46:05.273146 14613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:05.274772 14590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:46:05.275053 14590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "167bfe20ec0446c3b2968faf0b11ef4a"
format_stamp: "Formatted at 2026-08-12 16:46:05 on dist-test-slave-1vmg"
server_key: "1ac6551642f0ad485e1899bdb6415963"
server_key_iv: "20f60b199b31f301e982cb34df01e646"
server_key_version: "encryptionkey@0"
I20260812 16:46:05.275318 14590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:05.282714 14590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:05.283721 14590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:05.284010 14590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:05.285102 14590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:05.286823 14590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:05.286988 14590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:05.287117 14590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:05.287191 14590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:05.303869 14371 heartbeater.cc:503] Master 127.28.243.62:36453 was elected leader, sending a full tablet report...
I20260812 16:46:05.329017 14590 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:42725
I20260812 16:46:05.329074 14790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:42725 every 8 connection(s)
I20260812 16:46:05.330960 14590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:46:05.335644 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 14590
I20260812 16:46:05.335896 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:46:05.336613 29644 external_mini_cluster.cc:1428] Setting key 30ec7f3c68da87627432b3979c6b7349
I20260812 16:46:05.356643 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:05.341037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:36835 (local address 127.28.243.62:36453)
0812 16:46:05.351411 (+ 10374us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:05.351416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:05.352131 (+ 715us) server_negotiation.cc:407] Connection header received
0812 16:46:05.354037 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:05.354042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:05.354128 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:05.354246 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:05.355982 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:05.355987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:05.355990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:05.356009 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:05.356057 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:05.356070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:05.356191 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:05.356344 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 16:46:05.356398 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10230,"thread_start_us":10106,"threads_started":1}
I20260812 16:46:05.357676 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:05.339770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36453 (local address 127.28.243.3:36835)
0812 16:46:05.351897 (+ 12127us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:05.351959 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:05.353674 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:05.354635 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:05.354648 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:05.354799 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:05.355633 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:46:05.355636 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:05.355680 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:46:05.355777 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:05.356173 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:05.356179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:05.356185 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 16:46:05.356415 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 16:46:05.356648 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11483,"thread_start_us":14122,"threads_started":1}
I20260812 16:46:05.359481 14792 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36453
I20260812 16:46:05.359841 14792 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:05.360534 14792 heartbeater.cc:511] Master 127.28.243.62:36453 requested a full tablet report, sending...
I20260812 16:46:05.362138 14066 ts_manager.cc:194] Registered new tserver with Master: 167bfe20ec0446c3b2968faf0b11ef4a (127.28.243.3:42725)
I20260812 16:46:05.368050 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 16:46:05.376736 14799 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:46:05.374187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.28.243.62:36453)
0812 16:46:05.374640 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:05.374650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:05.375520 (+ 870us) server_negotiation.cc:407] Connection header received
0812 16:46:05.375754 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:05.375759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:05.375833 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:05.375939 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:05.376454 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45324: BlockingRecv error: recv got EOF from 127.0.0.1:45324 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:46:05.377727 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 14162
I20260812 16:46:05.384101 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:05.379289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.28.243.1:46135)
0812 16:46:05.380378 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:05.380388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:05.380435 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:46:05.380534 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:05.380543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:05.380911 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:05.381111 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:05.382793 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:05.382802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:05.382817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:05.382834 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:05.382881 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:05.382898 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:05.383160 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:05.383624 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 16:46:05.383760 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":686,"threads_started":1}
I20260812 16:46:05.385685 14364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:05.471230 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 14162
I20260812 16:46:05.495914 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 14377
I20260812 16:46:05.505726 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:05.496989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.28.243.2:42619)
0812 16:46:05.497520 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:05.497527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:05.497565 (+ 38us) server_negotiation.cc:407] Connection header received
0812 16:46:05.497634 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:05.497642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:05.497873 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:05.498040 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:05.504942 (+ 6902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:05.504955 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:05.504972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:05.504996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:05.505074 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:05.505098 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:05.505246 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:05.505344 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 16:46:05.505467 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260812 16:46:05.507227 14579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:05.590853 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 14377
I20260812 16:46:05.615960 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 14590
I20260812 16:46:05.621284 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:05.617065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.28.243.3:42725)
0812 16:46:05.617772 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:05.617782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:05.617822 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:46:05.617900 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:05.617907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:05.618137 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:05.618273 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:05.619715 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:05.619727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:05.619747 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:46:05.619787 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:05.619886 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:46:05.619913 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:05.620150 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:05.620579 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 16:46:05.620777 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":350,"threads_started":1}
I20260812 16:46:05.622977 14786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:05.708896 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 14590
I20260812 16:46:05.736148 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 14021
I20260812 16:46:05.737388 14109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:05.848187 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 14021
2026-08-12T16:46:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2893 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:46:05 dist-test-slave-1vmg krb5kdc[14827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:46:05 dist-test-slave-1vmg krb5kdc[14827](info): set up 2 sockets
Aug 12 16:46:05 dist-test-slave-1vmg krb5kdc[14827](info): commencing operation
krb5kdc: starting...
W20260812 16:46:07.204674 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.011s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:46:07 dist-test-slave-1vmg krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:46:07.567286 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43443
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:46:08.014439 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:08.014873 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:08.015576 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:08.025777 14850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:46:08.025954 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:08.026038 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:08.026127 14850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:46:08.026188 14850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:46:08.037799 14850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:08.039705 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:08.041987 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:08.058596 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:08.058670 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:08.059482 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:08.060735 14850 server_base.cc:1034] running on GCE node
I20260812 16:46:08.061693 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:08.063477 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:08.064760 14850 hybrid_clock.cc:648] HybridClock initialized: now 1786553168064704 us; error 51 us; skew 500 ppm
I20260812 16:46:08.065176 14850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:46:08 dist-test-slave-1vmg krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:46:08.068920 14850 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:46:08.074260 14850 webserver.cc:460] Webserver started at http://127.28.243.62:36585/ using document root <none> and password file <none>
I20260812 16:46:08.075456 14850 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:08.075629 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:08.076097 14850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:08.079725 14850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "2554424746fc42a9ad4aaa388a5dc49c"
format_stamp: "Formatted at 2026-08-12 16:46:08 on dist-test-slave-1vmg"
server_key: "946fbcb51f8b92d519265eb48180369c"
server_key_iv: "abf18adc07b66b0b8b1e788adf6b35f9"
server_key_version: "encryptionkey@0"
I20260812 16:46:08.080621 14850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "2554424746fc42a9ad4aaa388a5dc49c"
format_stamp: "Formatted at 2026-08-12 16:46:08 on dist-test-slave-1vmg"
server_key: "946fbcb51f8b92d519265eb48180369c"
server_key_iv: "abf18adc07b66b0b8b1e788adf6b35f9"
server_key_version: "encryptionkey@0"
I20260812 16:46:08.086603 14850 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 16:46:08.090526 14874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:08.092502 14850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 16:46:08.092751 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "2554424746fc42a9ad4aaa388a5dc49c"
format_stamp: "Formatted at 2026-08-12 16:46:08 on dist-test-slave-1vmg"
server_key: "946fbcb51f8b92d519265eb48180369c"
server_key_iv: "abf18adc07b66b0b8b1e788adf6b35f9"
server_key_version: "encryptionkey@0"
I20260812 16:46:08.093070 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:08.120167 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:08.125931 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:08.126286 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:08.148327 14850 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:43443
I20260812 16:46:08.148334 14957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:43443 every 8 connection(s)
I20260812 16:46:08.150372 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:46:08.156121 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:46:08.159343 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 14850
I20260812 16:46:08.159595 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:46:08.160282 29644 external_mini_cluster.cc:1428] Setting key be45969f35a1b8ff330c749eabaa1cb6
I20260812 16:46:08.172405 14958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c: Bootstrap starting.
I20260812 16:46:08.177920 14958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c: Neither blocks nor log segments found. Creating new log.
I20260812 16:46:08.179795 14958 log.cc:824] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:46:08 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:46:08.184206 14958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c: No bootstrap required, opened a new log
I20260812 16:46:08.192272 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:08.162738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.28.243.62:43443)
0812 16:46:08.163770 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:08.163787 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:08.163860 (+ 73us) server_negotiation.cc:407] Connection header received
0812 16:46:08.166301 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:08.166336 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:08.166951 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:08.167451 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:08.169576 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:08.171668 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:08.173800 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:08.175023 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:08.175261 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:08.181560 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:08.181574 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:08.181601 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:08.181663 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:08.185868 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:08.186803 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:08.186813 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:08.186824 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:08.187109 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:08.187500 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:08.187506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:08.187509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:08.188342 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:08.188703 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:08.189193 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 16:46:08.189752 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":464,"threads_started":1}
I20260812 16:46:08.195932 14958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2554424746fc42a9ad4aaa388a5dc49c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43443 } }
I20260812 16:46:08.196414 14958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:46:08.196521 14958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2554424746fc42a9ad4aaa388a5dc49c, State: Initialized, Role: FOLLOWER
I20260812 16:46:08.197389 14958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2554424746fc42a9ad4aaa388a5dc49c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43443 } }
I20260812 16:46:08.197827 14958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:46:08.198020 14958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:46:08.198249 14958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:46:08.201067 14958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2554424746fc42a9ad4aaa388a5dc49c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43443 } }
I20260812 16:46:08.201786 14958 leader_election.cc:302] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2554424746fc42a9ad4aaa388a5dc49c; no voters:
I20260812 16:46:08.202476 14958 leader_election.cc:288] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:46:08.202651 14967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:46:08.203431 14967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [term 1 LEADER]: Becoming Leader. State: Replica: 2554424746fc42a9ad4aaa388a5dc49c, State: Running, Role: LEADER
I20260812 16:46:08.204030 14967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2554424746fc42a9ad4aaa388a5dc49c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43443 } }
I20260812 16:46:08.204766 14958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:46:08.207203 14971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2554424746fc42a9ad4aaa388a5dc49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2554424746fc42a9ad4aaa388a5dc49c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43443 } } }
I20260812 16:46:08.207252 14973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2554424746fc42a9ad4aaa388a5dc49c. Latest consensus state: current_term: 1 leader_uuid: "2554424746fc42a9ad4aaa388a5dc49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2554424746fc42a9ad4aaa388a5dc49c" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43443 } } }
I20260812 16:46:08.207604 14971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [sys.catalog]: This master's current role is: LEADER
I20260812 16:46:08.207638 14973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c [sys.catalog]: This master's current role is: LEADER
I20260812 16:46:08.213155 14978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:46:08.221474 14978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:46:08.235886 14978 catalog_manager.cc:1347] Generated new cluster ID: e5697256299c428fbbb28d34506b0ee5
I20260812 16:46:08.236107 14978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:46:08.247645 14978 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:46:08.250001 14978 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:46:08.264465 14978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c: Generated new TSK 0
I20260812 16:46:08.265726 14978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:46:08.359582 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43443
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:46:08.774115 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:08.774466 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:08.774571 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:08.782804 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:08.783013 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:08.783143 15002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:46:08.794116 15002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:08.795624 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:08.797020 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:08.811913 15010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:08.811930 15009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:08.811954 15013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:08.812809 15002 server_base.cc:1034] running on GCE node
I20260812 16:46:08.813503 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:08.814596 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:08.815865 15002 hybrid_clock.cc:648] HybridClock initialized: now 1786553168815806 us; error 56 us; skew 500 ppm
I20260812 16:46:08.816243 15002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:46:08 dist-test-slave-1vmg krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:46:08.819875 15002 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:46:08.822643 15002 webserver.cc:460] Webserver started at http://127.28.243.1:39821/ using document root <none> and password file <none>
I20260812 16:46:08.823596 15002 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:08.823740 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:08.824082 15002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:08.827515 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "19d3933eeb0d46f598e3c1657958ca33"
format_stamp: "Formatted at 2026-08-12 16:46:08 on dist-test-slave-1vmg"
server_key: "0e5b5ed42060b447ffc538a836359626"
server_key_iv: "703fb5c53fef4d26166a0f27ed17eab3"
server_key_version: "encryptionkey@0"
I20260812 16:46:08.828362 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "19d3933eeb0d46f598e3c1657958ca33"
format_stamp: "Formatted at 2026-08-12 16:46:08 on dist-test-slave-1vmg"
server_key: "0e5b5ed42060b447ffc538a836359626"
server_key_iv: "703fb5c53fef4d26166a0f27ed17eab3"
server_key_version: "encryptionkey@0"
I20260812 16:46:08.834458 15002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:46:08.838632 15024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:08.840162 15002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:46:08.840346 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "19d3933eeb0d46f598e3c1657958ca33"
format_stamp: "Formatted at 2026-08-12 16:46:08 on dist-test-slave-1vmg"
server_key: "0e5b5ed42060b447ffc538a836359626"
server_key_iv: "703fb5c53fef4d26166a0f27ed17eab3"
server_key_version: "encryptionkey@0"
I20260812 16:46:08.840559 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:08.861887 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:08.866536 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:08.866847 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:08.867900 15002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:08.869336 15002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:08.869423 15002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:08.869557 15002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:08.869611 15002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:08.904122 15002 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:32993
I20260812 16:46:08.904184 15193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:32993 every 8 connection(s)
I20260812 16:46:08.905647 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:46:08.910396 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 15002
I20260812 16:46:08.910651 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:46:08.911340 29644 external_mini_cluster.cc:1428] Setting key 247174fe0a4a9e6dd5ef12821c1fbc0c
Aug 12 16:46:08 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:46:08.931552 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:08.908509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:42775 (local address 127.28.243.62:43443)
0812 16:46:08.909037 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:08.909046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:08.910214 (+ 1168us) server_negotiation.cc:407] Connection header received
0812 16:46:08.912289 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:08.912295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:08.912394 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:08.912513 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:08.914748 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:08.915567 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:08.918046 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:08.918498 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:08.918619 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:08.923868 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:08.923875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:08.923878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:08.923924 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:08.927742 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:08.928868 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:08.928875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:08.928879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:08.929055 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:08.929543 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:08.929549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:08.929552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:08.929912 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:08.930068 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:08.930909 (+ 841us) server_negotiation.cc:299] Negotiation successful
0812 16:46:08.931200 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260812 16:46:08.932559 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:08.909096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43443 (local address 127.28.243.1:42775)
0812 16:46:08.909900 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:08.910041 (+ 141us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:08.911960 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:08.912780 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:08.912790 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:08.913566 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:08.914490 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:08.914521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:08.915768 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:08.915774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:08.917760 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:08.917783 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:08.918788 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:08.918795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:08.919157 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:08.920355 (+ 1198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:46:08.920397 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:08.923475 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:08.928110 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:46:08.928120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:46:08.928137 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:46:08.928696 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:46:08.929205 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:46:08.929209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:46:08.929212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:46:08.929426 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:46:08.930076 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:08.930085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:08.930488 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 16:46:08.930770 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 16:46:08.931314 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":308,"threads_started":1}
I20260812 16:46:08.934546 15194 heartbeater.cc:347] Connected to a master server at 127.28.243.62:43443
I20260812 16:46:08.934980 15194 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:08.936020 15194 heartbeater.cc:511] Master 127.28.243.62:43443 requested a full tablet report, sending...
I20260812 16:46:08.938509 14905 ts_manager.cc:194] Registered new tserver with Master: 19d3933eeb0d46f598e3c1657958ca33 (127.28.243.1:32993)
I20260812 16:46:08.940130 14905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:42775
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:46:08.982656 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43443
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:46:09.361769 15202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:09.362279 15202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:09.362432 15202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:09.372275 15202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:09.372468 15202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:09.372645 15202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:46:09.385260 15202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:09.386708 15202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:09.388156 15202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:09.404217 15209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:09.404254 15212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:09.404446 15216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:09.405469 15202 server_base.cc:1034] running on GCE node
I20260812 16:46:09.406561 15202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:09.408080 15202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:09.409406 15202 hybrid_clock.cc:648] HybridClock initialized: now 1786553169409354 us; error 42 us; skew 500 ppm
I20260812 16:46:09.409880 15202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:46:09 dist-test-slave-1vmg krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:46:09.414598 15202 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:46:09.416572 15202 webserver.cc:460] Webserver started at http://127.28.243.2:32827/ using document root <none> and password file <none>
I20260812 16:46:09.417392 15202 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:09.417537 15202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:09.417869 15202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:09.422245 15202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "a1f061e088c440f5aea340a1e5d2d42d"
format_stamp: "Formatted at 2026-08-12 16:46:09 on dist-test-slave-1vmg"
server_key: "8843419d2e71d3cfba693d9a3422c5bc"
server_key_iv: "75c4cf999e1a11cfc0576e37c7a113e8"
server_key_version: "encryptionkey@0"
I20260812 16:46:09.423791 15202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "a1f061e088c440f5aea340a1e5d2d42d"
format_stamp: "Formatted at 2026-08-12 16:46:09 on dist-test-slave-1vmg"
server_key: "8843419d2e71d3cfba693d9a3422c5bc"
server_key_iv: "75c4cf999e1a11cfc0576e37c7a113e8"
server_key_version: "encryptionkey@0"
I20260812 16:46:09.430833 15202 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:46:09.435703 15230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:09.438237 15202 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:46:09.438488 15202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "a1f061e088c440f5aea340a1e5d2d42d"
format_stamp: "Formatted at 2026-08-12 16:46:09 on dist-test-slave-1vmg"
server_key: "8843419d2e71d3cfba693d9a3422c5bc"
server_key_iv: "75c4cf999e1a11cfc0576e37c7a113e8"
server_key_version: "encryptionkey@0"
I20260812 16:46:09.438730 15202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:09.454384 15202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:09.459456 15202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:09.459832 15202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:09.461248 15202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:09.463557 15202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:09.463680 15202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:09.463811 15202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:09.463874 15202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:09.501000 15202 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:43395
I20260812 16:46:09.501060 15404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:43395 every 8 connection(s)
I20260812 16:46:09.502506 15202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:46:09.508508 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 15202
I20260812 16:46:09.508713 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:46:09.509138 29644 external_mini_cluster.cc:1428] Setting key a2696bb7045bf9e5904317b01e08ef96
Aug 12 16:46:09 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:46:09.532554 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:09.505812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:58775 (local address 127.28.243.62:43443)
0812 16:46:09.506444 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:09.506449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:09.508066 (+ 1617us) server_negotiation.cc:407] Connection header received
0812 16:46:09.510526 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:09.510530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:09.510602 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:09.510700 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:09.514696 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:09.515859 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:09.519010 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:09.519746 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:09.520117 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:09.524727 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:09.524733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:09.524736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:09.524779 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:09.527758 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:09.528695 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:09.528700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:09.528702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:09.528856 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:09.529793 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:09.529799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:09.529801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:09.530278 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:09.530460 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:09.531599 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0812 16:46:09.532158 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":327,"threads_started":1}
I20260812 16:46:09.533742 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:09.506670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43443 (local address 127.28.243.2:58775)
0812 16:46:09.507829 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:09.507889 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:09.510067 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:09.511212 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:09.511227 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:09.512700 (+ 1473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:09.514278 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:09.514382 (+ 104us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:09.516243 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:09.516251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:09.518600 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:09.518628 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:09.520230 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:09.520240 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:09.520650 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:09.521657 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:46:09.521696 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:09.524382 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:09.527983 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:46:09.527993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:46:09.528010 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:46:09.528532 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:46:09.528994 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:46:09.529000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:46:09.529003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:46:09.529522 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:46:09.530483 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:09.530496 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:09.531093 (+ 597us) client_negotiation.cc:769] Sending connection context
0812 16:46:09.531434 (+ 341us) client_negotiation.cc:240] Negotiation successful
0812 16:46:09.532384 (+ 950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":563,"threads_started":1}
I20260812 16:46:09.535501 15405 heartbeater.cc:347] Connected to a master server at 127.28.243.62:43443
I20260812 16:46:09.535858 15405 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:09.537004 15405 heartbeater.cc:511] Master 127.28.243.62:43443 requested a full tablet report, sending...
I20260812 16:46:09.539436 14905 ts_manager.cc:194] Registered new tserver with Master: a1f061e088c440f5aea340a1e5d2d42d (127.28.243.2:43395)
I20260812 16:46:09.540386 14905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:58775
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:46:09.592921 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43443
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:46:09.944203 15194 heartbeater.cc:503] Master 127.28.243.62:43443 was elected leader, sending a full tablet report...
W20260812 16:46:09.997828 15412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:46:09.998168 15412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:46:09.998274 15412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:46:10.008445 15412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:46:10.008589 15412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:46:10.008747 15412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:46:10.020076 15412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:46:10.023443 15412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:46:10.025166 15412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:46:10.045910 15423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:10.046665 15428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:46:10.049191 15424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:46:10.049206 15412 server_base.cc:1034] running on GCE node
I20260812 16:46:10.051465 15412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:46:10.053527 15412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:46:10.054903 15412 hybrid_clock.cc:648] HybridClock initialized: now 1786553170054828 us; error 75 us; skew 500 ppm
I20260812 16:46:10.055595 15412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:46:10 dist-test-slave-1vmg krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553170, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:46:10.060674 15412 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:46:10.062815 15412 webserver.cc:460] Webserver started at http://127.28.243.3:41077/ using document root <none> and password file <none>
I20260812 16:46:10.063900 15412 fs_manager.cc:359] Metadata directory not provided
I20260812 16:46:10.064105 15412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:46:10.064503 15412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:46:10.068055 15412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "d8223794bef74026a5cd016aa000e2ff"
format_stamp: "Formatted at 2026-08-12 16:46:10 on dist-test-slave-1vmg"
server_key: "5495db3a7eec5d921d4532800d66678d"
server_key_iv: "926060bc450a647ce202ea20e7bb65d4"
server_key_version: "encryptionkey@0"
I20260812 16:46:10.068986 15412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "d8223794bef74026a5cd016aa000e2ff"
format_stamp: "Formatted at 2026-08-12 16:46:10 on dist-test-slave-1vmg"
server_key: "5495db3a7eec5d921d4532800d66678d"
server_key_iv: "926060bc450a647ce202ea20e7bb65d4"
server_key_version: "encryptionkey@0"
I20260812 16:46:10.078604 15412 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260812 16:46:10.085047 15436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:10.086961 15412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260812 16:46:10.087210 15412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "d8223794bef74026a5cd016aa000e2ff"
format_stamp: "Formatted at 2026-08-12 16:46:10 on dist-test-slave-1vmg"
server_key: "5495db3a7eec5d921d4532800d66678d"
server_key_iv: "926060bc450a647ce202ea20e7bb65d4"
server_key_version: "encryptionkey@0"
I20260812 16:46:10.087460 15412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:46:10.114799 15412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:46:10.122015 15412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:46:10.122376 15412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:46:10.123585 15412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:46:10.125327 15412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:46:10.125418 15412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:10.125550 15412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:46:10.125607 15412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:46:10.168625 15412 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:34539
I20260812 16:46:10.168730 15619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:34539 every 8 connection(s)
I20260812 16:46:10.170161 15412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:46:10.177301 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 15412
I20260812 16:46:10.177534 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:46:10.178081 29644 external_mini_cluster.cc:1428] Setting key 7ebff11054c677b8376f18aa274c4da7
Aug 12 16:46:10 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553170, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:46:10.197819 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.173595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:48921 (local address 127.28.243.62:43443)
0812 16:46:10.174266 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.174271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.176201 (+ 1930us) server_negotiation.cc:407] Connection header received
0812 16:46:10.178908 (+ 2707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.178933 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.179042 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.179147 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:10.181083 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.181926 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.184771 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.185286 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.185434 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.189922 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:10.189929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:10.189933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:10.189992 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:10.193210 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.194276 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.194281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.194285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.194490 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.195126 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.195133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.195137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.195789 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:10.196014 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.197082 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.197428 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":286,"threads_started":1}
I20260812 16:46:10.198652 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.174781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43443 (local address 127.28.243.3:48921)
0812 16:46:10.175983 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:10.176035 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:10.178535 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:10.179457 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:10.179468 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:10.180066 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:46:10.180896 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.180918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.182101 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.182105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.184402 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.184439 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.185743 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.185769 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.186288 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.187111 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:46:10.187139 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:46:10.189568 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:46:10.193414 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:46:10.193426 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:46:10.193446 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:46:10.194130 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:46:10.194617 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:46:10.194622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:46:10.194625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:46:10.194908 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:46:10.196025 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:46:10.196033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:46:10.196476 (+ 443us) client_negotiation.cc:769] Sending connection context
0812 16:46:10.196761 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 16:46:10.197405 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":444,"threads_started":1}
I20260812 16:46:10.200985 15620 heartbeater.cc:347] Connected to a master server at 127.28.243.62:43443
I20260812 16:46:10.201531 15620 heartbeater.cc:464] Registering TS with master...
I20260812 16:46:10.202378 15620 heartbeater.cc:511] Master 127.28.243.62:43443 requested a full tablet report, sending...
I20260812 16:46:10.204512 14905 ts_manager.cc:194] Registered new tserver with Master: d8223794bef74026a5cd016aa000e2ff (127.28.243.3:34539)
I20260812 16:46:10.205943 14905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:48921
I20260812 16:46:10.208294 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:46:10.233634 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.214730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.28.243.62:43443)
0812 16:46:10.215132 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.215138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.215776 (+ 638us) server_negotiation.cc:407] Connection header received
0812 16:46:10.216174 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.216179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.216268 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.216384 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:10.218865 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.219874 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.221511 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.222218 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.222404 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.225432 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:10.225438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:10.225442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:10.225501 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:10.229270 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.230776 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.230781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.230784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.231122 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.231629 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.231632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.231635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.232032 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:10.232259 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.233090 (+ 831us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.233353 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260812 16:46:10.241065 14905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:46:10.248648 14905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:46:10.277140 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.264919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.28.243.2:43395)
0812 16:46:10.267430 (+ 2511us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.267439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.267472 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:46:10.267581 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.267589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.267914 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.269026 (+ 1112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:46:10.269289 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.271713 (+ 2424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.274524 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.275826 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.275974 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.276193 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.276593 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.276787 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":2207,"threads_started":1}
I20260812 16:46:10.277184 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.264807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:43395 (local address 127.0.0.1:44660)
0812 16:46:10.266255 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:10.266311 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:10.266547 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:10.268216 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:10.268226 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:10.268265 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:46:10.268784 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.268799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.271953 (+ 3154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.271959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.274274 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.274294 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.276054 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.276060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.276357 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.276400 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 16:46:10.276526 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 16:46:10.276766 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":107,"thread_start_us":2771,"threads_started":1}
I20260812 16:46:10.279918 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.265374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:34539 (local address 127.0.0.1:47578)
0812 16:46:10.267884 (+ 2510us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:10.267904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:10.268029 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:10.270329 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:10.270334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:10.270363 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:46:10.270823 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.270836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.272964 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.272969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.275846 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.275863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.278668 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.278676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.278864 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.279036 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 16:46:10.279125 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 16:46:10.279308 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2385,"mutex_wait_us":37,"thread_start_us":1088,"threads_started":1}
I20260812 16:46:10.281212 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.266133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.28.243.3:34539)
0812 16:46:10.269681 (+ 3548us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.269689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.269739 (+ 50us) server_negotiation.cc:407] Connection header received
0812 16:46:10.269831 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.269837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.270078 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.271090 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:46:10.271388 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.272763 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.276126 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.278362 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.280202 (+ 1840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.280518 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.280613 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.280856 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3338,"thread_start_us":3068,"threads_started":1}
I20260812 16:46:10.281637 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.269056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:32993 (local address 127.0.0.1:45284)
0812 16:46:10.270437 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:10.270464 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:10.270710 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:10.272039 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:10.272043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:10.272084 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:46:10.272595 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.272606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.274908 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.274949 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.277791 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.277891 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.280731 (+ 2840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.280738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.280930 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.280973 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 16:46:10.281099 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 16:46:10.281279 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":4,"thread_start_us":533,"threads_started":1}
I20260812 16:46:10.282125 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.268365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.28.243.1:32993)
0812 16:46:10.269011 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.269020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.270586 (+ 1566us) server_negotiation.cc:407] Connection header received
0812 16:46:10.270965 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.270975 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.271213 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.271367 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:46:10.272823 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.274697 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.278186 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.280080 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.280266 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.280560 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.281190 (+ 630us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.281422 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":339,"threads_started":1}
I20260812 16:46:10.283602 15302 tablet_service.cc:1467] Processing CreateTablet for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac (DEFAULT_TABLE table=test-table [id=6b0be6c9d0bd4cea982f3ec0cf89ea70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:46:10.285499 15302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:46:10.286332 15513 tablet_service.cc:1467] Processing CreateTablet for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac (DEFAULT_TABLE table=test-table [id=6b0be6c9d0bd4cea982f3ec0cf89ea70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:46:10.286652 15099 tablet_service.cc:1467] Processing CreateTablet for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac (DEFAULT_TABLE table=test-table [id=6b0be6c9d0bd4cea982f3ec0cf89ea70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:46:10.287801 15513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:46:10.288190 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:46:10.303097 15653 tablet_bootstrap.cc:492] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Bootstrap starting.
I20260812 16:46:10.305164 15656 tablet_bootstrap.cc:492] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Bootstrap starting.
I20260812 16:46:10.308733 15653 tablet_bootstrap.cc:654] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:46:10.308945 15656 tablet_bootstrap.cc:654] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Neither blocks nor log segments found. Creating new log.
I20260812 16:46:10.310484 15656 log.cc:824] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Log is configured to *not* fsync() on all Append() calls
I20260812 16:46:10.310767 15654 tablet_bootstrap.cc:492] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Bootstrap starting.
I20260812 16:46:10.311542 15653 log.cc:824] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Log is configured to *not* fsync() on all Append() calls
I20260812 16:46:10.315261 15656 tablet_bootstrap.cc:492] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: No bootstrap required, opened a new log
I20260812 16:46:10.315572 15656 ts_tablet_manager.cc:1397] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260812 16:46:10.316138 15653 tablet_bootstrap.cc:492] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: No bootstrap required, opened a new log
I20260812 16:46:10.316448 15653 ts_tablet_manager.cc:1397] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20260812 16:46:10.316948 15654 tablet_bootstrap.cc:654] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Neither blocks nor log segments found. Creating new log.
I20260812 16:46:10.318383 15654 log.cc:824] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:46:10.323099 15654 tablet_bootstrap.cc:492] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: No bootstrap required, opened a new log
I20260812 16:46:10.323482 15654 ts_tablet_manager.cc:1397] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260812 16:46:10.325368 15653 raft_consensus.cc:348] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.325790 15653 raft_consensus.cc:374] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:46:10.325909 15653 raft_consensus.cc:729] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8223794bef74026a5cd016aa000e2ff, State: Initialized, Role: FOLLOWER
I20260812 16:46:10.325690 15656 raft_consensus.cc:348] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.326196 15656 raft_consensus.cc:374] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:46:10.326350 15656 raft_consensus.cc:729] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d3933eeb0d46f598e3c1657958ca33, State: Initialized, Role: FOLLOWER
I20260812 16:46:10.326752 15653 consensus_queue.cc:260] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.327020 15656 consensus_queue.cc:260] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.328095 15620 heartbeater.cc:503] Master 127.28.243.62:43443 was elected leader, sending a full tablet report...
I20260812 16:46:10.328969 15653 ts_tablet_manager.cc:1428] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260812 16:46:10.330871 15656 ts_tablet_manager.cc:1428] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260812 16:46:10.335855 15654 raft_consensus.cc:348] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.336282 15654 raft_consensus.cc:374] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:46:10.336375 15654 raft_consensus.cc:729] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f061e088c440f5aea340a1e5d2d42d, State: Initialized, Role: FOLLOWER
I20260812 16:46:10.336994 15654 consensus_queue.cc:260] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.338424 15405 heartbeater.cc:503] Master 127.28.243.62:43443 was elected leader, sending a full tablet report...
I20260812 16:46:10.338697 15654 ts_tablet_manager.cc:1428] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260812 16:46:10.408742 15196 tablet.cc:2369] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:46:10.422813 15621 tablet.cc:2369] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:46:10.427321 15660 raft_consensus.cc:482] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:46:10.427686 15660 raft_consensus.cc:504] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.429747 15660 leader_election.cc:288] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1f061e088c440f5aea340a1e5d2d42d (127.28.243.2:43395), 19d3933eeb0d46f598e3c1657958ca33 (127.28.243.1:32993)
I20260812 16:46:10.438601 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.430149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:43395 (local address 127.28.243.3:47149)
0812 16:46:10.430369 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:10.430391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:10.430560 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:10.431338 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:10.431343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:10.431377 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:46:10.431598 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.431610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.433360 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.433366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.435712 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.435734 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.437456 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.437461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.437659 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.437697 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:46:10.437986 (+ 289us) client_negotiation.cc:240] Negotiation successful
0812 16:46:10.438180 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 16:46:10.438660 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.430507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:47149 (local address 127.28.243.2:43395)
0812 16:46:10.430837 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.430844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.430876 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:46:10.431029 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.431036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.431142 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.431288 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:46:10.431815 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.433069 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.435959 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.437190 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.437378 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.437463 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.438125 (+ 662us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.438321 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 16:46:10.438967 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.431042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:50697 (local address 127.28.243.1:32993)
0812 16:46:10.431287 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.431293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.431454 (+ 161us) server_negotiation.cc:407] Connection header received
0812 16:46:10.431732 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.431739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.431834 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.432103 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:46:10.432653 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.433718 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.436487 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.437665 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.437811 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.437871 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.438482 (+ 611us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.438678 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:46:10.439347 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.430636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:32993 (local address 127.28.243.3:50697)
0812 16:46:10.431338 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0812 16:46:10.431357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:46:10.431488 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:46:10.432141 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:46:10.432145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:46:10.432180 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:46:10.432470 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.432479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.433950 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.433956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.436251 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:46:10.436265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.437987 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.437995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:46:10.438192 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.438238 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 16:46:10.438349 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 16:46:10.438600 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":359,"threads_started":1}
I20260812 16:46:10.440639 15332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7eff9e7fd74ce2a7a6473b117dd7ac" candidate_uuid: "d8223794bef74026a5cd016aa000e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f061e088c440f5aea340a1e5d2d42d" is_pre_election: true
I20260812 16:46:10.440673 15128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7eff9e7fd74ce2a7a6473b117dd7ac" candidate_uuid: "d8223794bef74026a5cd016aa000e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d3933eeb0d46f598e3c1657958ca33" is_pre_election: true
I20260812 16:46:10.441207 15332 raft_consensus.cc:2454] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8223794bef74026a5cd016aa000e2ff in term 0.
I20260812 16:46:10.441416 15128 raft_consensus.cc:2454] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8223794bef74026a5cd016aa000e2ff in term 0.
I20260812 16:46:10.442056 15438 leader_election.cc:302] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1f061e088c440f5aea340a1e5d2d42d, d8223794bef74026a5cd016aa000e2ff; no voters:
I20260812 16:46:10.442637 15660 raft_consensus.cc:2781] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:46:10.442790 15660 raft_consensus.cc:482] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:46:10.442951 15660 raft_consensus.cc:3037] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:46:10.445370 15660 raft_consensus.cc:504] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.446041 15660 leader_election.cc:288] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [CANDIDATE]: Term 1 election: Requested vote from peers a1f061e088c440f5aea340a1e5d2d42d (127.28.243.2:43395), 19d3933eeb0d46f598e3c1657958ca33 (127.28.243.1:32993)
I20260812 16:46:10.446942 15332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7eff9e7fd74ce2a7a6473b117dd7ac" candidate_uuid: "d8223794bef74026a5cd016aa000e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f061e088c440f5aea340a1e5d2d42d"
I20260812 16:46:10.447181 15332 raft_consensus.cc:3037] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:46:10.447180 15128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7eff9e7fd74ce2a7a6473b117dd7ac" candidate_uuid: "d8223794bef74026a5cd016aa000e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d3933eeb0d46f598e3c1657958ca33"
I20260812 16:46:10.447610 15128 raft_consensus.cc:3037] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:46:10.449651 15332 raft_consensus.cc:2454] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8223794bef74026a5cd016aa000e2ff in term 1.
I20260812 16:46:10.450095 15128 raft_consensus.cc:2454] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8223794bef74026a5cd016aa000e2ff in term 1.
I20260812 16:46:10.450182 15438 leader_election.cc:302] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1f061e088c440f5aea340a1e5d2d42d, d8223794bef74026a5cd016aa000e2ff; no voters:
I20260812 16:46:10.450520 15660 raft_consensus.cc:2781] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:46:10.451042 15660 raft_consensus.cc:686] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 1 LEADER]: Becoming Leader. State: Replica: d8223794bef74026a5cd016aa000e2ff, State: Running, Role: LEADER
I20260812 16:46:10.451560 15660 consensus_queue.cc:237] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } }
I20260812 16:46:10.457100 14904 catalog_manager.cc:5495] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to d8223794bef74026a5cd016aa000e2ff (127.28.243.3). New cstate: current_term: 1 leader_uuid: "d8223794bef74026a5cd016aa000e2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8223794bef74026a5cd016aa000e2ff" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34539 } health_report { overall_health: HEALTHY } } }
I20260812 16:46:10.479096 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.473884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.28.243.3:34539)
0812 16:46:10.474082 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.474086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.474105 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:46:10.474307 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.474310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.474390 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.474521 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:46:10.474833 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.475610 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.476946 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.477355 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.477450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.477950 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:46:10.478484 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:46:10.478591 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.478752 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.478829 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:46:10.488830 15332 raft_consensus.cc:1261] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 1 FOLLOWER]: Refusing update from remote peer d8223794bef74026a5cd016aa000e2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:46:10.488826 15128 raft_consensus.cc:1261] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 1 FOLLOWER]: Refusing update from remote peer d8223794bef74026a5cd016aa000e2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:46:10.489948 15660 consensus_queue.cc:1035] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1f061e088c440f5aea340a1e5d2d42d" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:46:10.490439 15666 consensus_queue.cc:1035] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d3933eeb0d46f598e3c1657958ca33" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:46:10.505354 15407 tablet.cc:2369] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:46:10.537461 14904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:46:10.537829 14904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:46:10.542078 14904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2554424746fc42a9ad4aaa388a5dc49c: Sending DeleteTablet for 3 replicas of tablet fa7eff9e7fd74ce2a7a6473b117dd7ac
I20260812 16:46:10.543566 15302 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:46:10 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:44660
I20260812 16:46:10.543571 15513 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:46:10 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:47578
I20260812 16:46:10.543622 15099 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7eff9e7fd74ce2a7a6473b117dd7ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:46:10 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:45284
I20260812 16:46:10.547004 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 15002
I20260812 16:46:10.553610 15689 tablet_replica.cc:330] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: stopping tablet replica
I20260812 16:46:10.554399 15689 raft_consensus.cc:2229] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:46:10.555208 15689 raft_consensus.cc:2258] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:46:10.557977 15689 ts_tablet_manager.cc:1905] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:46:10.558702 15686 tablet_replica.cc:330] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: stopping tablet replica
I20260812 16:46:10.559406 15688 tablet_replica.cc:330] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: stopping tablet replica
I20260812 16:46:10.560019 15688 raft_consensus.cc:2229] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:46:10.560561 15686 raft_consensus.cc:2229] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:46:10.561216 15686 raft_consensus.cc:2258] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:46:10.562528 15688 raft_consensus.cc:2258] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:46:10.565683 15686 ts_tablet_manager.cc:1905] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:46:10.566803 15689 ts_tablet_manager.cc:1918] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:46:10.567179 15689 log.cc:1196] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/fa7eff9e7fd74ce2a7a6473b117dd7ac
I20260812 16:46:10.567920 15689 ts_tablet_manager.cc:1939] T fa7eff9e7fd74ce2a7a6473b117dd7ac P 19d3933eeb0d46f598e3c1657958ca33: Deleting consensus metadata
I20260812 16:46:10.568538 15688 ts_tablet_manager.cc:1905] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:46:10.570276 14875 catalog_manager.cc:4841] TS 19d3933eeb0d46f598e3c1657958ca33 (127.28.243.1:32993): tablet fa7eff9e7fd74ce2a7a6473b117dd7ac (table test-table [id=6b0be6c9d0bd4cea982f3ec0cf89ea70]) successfully deleted
Aug 12 16:46:10 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:46:10.575769 15686 ts_tablet_manager.cc:1918] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:46:10.576169 15686 log.cc:1196] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/fa7eff9e7fd74ce2a7a6473b117dd7ac
I20260812 16:46:10.576601 15688 ts_tablet_manager.cc:1918] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:46:10.576869 15686 ts_tablet_manager.cc:1939] T fa7eff9e7fd74ce2a7a6473b117dd7ac P a1f061e088c440f5aea340a1e5d2d42d: Deleting consensus metadata
I20260812 16:46:10.576860 15688 log.cc:1196] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/fa7eff9e7fd74ce2a7a6473b117dd7ac
I20260812 16:46:10.577525 15688 ts_tablet_manager.cc:1939] T fa7eff9e7fd74ce2a7a6473b117dd7ac P d8223794bef74026a5cd016aa000e2ff: Deleting consensus metadata
I20260812 16:46:10.579176 14875 catalog_manager.cc:4841] TS a1f061e088c440f5aea340a1e5d2d42d (127.28.243.2:43395): tablet fa7eff9e7fd74ce2a7a6473b117dd7ac (table test-table [id=6b0be6c9d0bd4cea982f3ec0cf89ea70]) successfully deleted
I20260812 16:46:10.579427 14876 catalog_manager.cc:4841] TS d8223794bef74026a5cd016aa000e2ff (127.28.243.3:34539): tablet fa7eff9e7fd74ce2a7a6473b117dd7ac (table test-table [id=6b0be6c9d0bd4cea982f3ec0cf89ea70]) successfully deleted
I20260812 16:46:10.583757 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.549101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.28.243.1:32993)
0812 16:46:10.549346 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.549354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.559067 (+ 9713us) server_negotiation.cc:407] Connection header received
0812 16:46:10.559384 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.559390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.559484 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.559563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:10.561435 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.562625 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.564531 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.565278 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.565400 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.576460 (+ 11060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:10.576488 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:10.576513 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:10.576582 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:10.580580 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.581439 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.581447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.581450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.581608 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.582005 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.582009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.582011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.582434 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:10.582757 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.583117 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.583366 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:46:10.584904 15185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:10.679613 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 15002
I20260812 16:46:10.717217 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 15202
Aug 12 16:46:10 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:46:10.737443 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.718088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.28.243.2:43395)
0812 16:46:10.718281 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.718286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.718306 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:46:10.718448 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.718452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.718521 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.718605 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:10.721219 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.722263 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.724498 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.724959 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.725105 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.730884 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:10.730902 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:10.730957 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:10.731021 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:10.734265 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.735165 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.735173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.735176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.735333 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.735741 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.735745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.735747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.736182 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:10.736481 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.736858 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.737101 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 16:46:10.738134 15393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:10.833663 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 15202
I20260812 16:46:10.860023 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 15412
Aug 12 16:46:10 dist-test-slave-1vmg krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:46:10.878604 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:46:10.861256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.28.243.3:34539)
0812 16:46:10.861570 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 16:46:10.861577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:46:10.861603 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:46:10.861696 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:46:10.861703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:46:10.861812 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:46:10.861978 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:46:10.863425 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.864597 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.865963 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:46:10.866658 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:46:10.866814 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:46:10.871989 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:46:10.872009 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:46:10.872029 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:46:10.872080 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:46:10.875272 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.876301 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.876310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.876312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.876513 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:46:10.876976 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:46:10.876980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:46:10.876982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:46:10.877399 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:46:10.877691 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 16:46:10.878062 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 16:46:10.878258 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 16:46:10.879683 15613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:11.013916 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 15412
I20260812 16:46:11.043756 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 14850
I20260812 16:46:11.045027 14952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:46:11.160852 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 14850
2026-08-12T16:46:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5314 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:46:11 dist-test-slave-1vmg krb5kdc[15711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:46:11 dist-test-slave-1vmg krb5kdc[15711](info): set up 2 sockets
Aug 12 16:46:11 dist-test-slave-1vmg krb5kdc[15711](info): commencing operation
krb5kdc: starting...
W20260812 16:46:12.516336 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:46:12 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:46:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/rangeradmin_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/rangeradmin_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/rangerlookup_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/rangerlookup_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
I20260812 16:46:13.084683 29644 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWMmoHF/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/postgres -l logfile start
2026-08-12 16:46:16.324 UTC [15762] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 16:46:16.324 UTC [15762] LOG: listening on IPv4 address "127.28.243.20", port 42975
2026-08-12 16:46:16.329 UTC [15762] LOG: listening on Unix socket "/tmp/.s.PGSQL.42975"
2026-08-12 16:46:16.356 UTC [15765] LOG: database system was shut down at 2026-08-12 16:46:14 UTC
2026-08-12 16:46:16.362 UTC [15762] LOG: database system is ready to accept connections
I20260812 16:46:17.555188 29644 mini_postgres.cc:96] Postgres bound to 42975
2026-08-12 16:46:17.584 UTC [15777] FATAL: database "slave" does not exist
127.28.243.20:42975 - accepting connections
I20260812 16:46:17.585544 29644 mini_ranger.cc:161] Starting Ranger...
I20260812 16:46:17.628144 29644 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 16:46:18.320397 29644 mini_ranger.cc:87] Created ranger Postgres database
I20260812 16:46:18.320538 29644 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-admin
2026-08-12 16:46:18,789 [I] DB FLAVOR :POSTGRES
2026-08-12 16:46:18,790 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:46:18,790 [I] Checking connection..
2026-08-12 16:46:18,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 16:46:19,117 [I] Checking connection passed.
2026-08-12 16:46:19,117 [I] --------- Verifying version history table ---------
2026-08-12 16:46:19,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:46:19,415 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:46:19,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:46:19,695 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:46:19,696 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 16:46:19,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 16:46:20.005 UTC [15885] WARNING: there is no transaction in progress
2026-08-12 16:46:20,015 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 16:46:20,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:46:20,318 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 16:46:20,318 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:46:20,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 16:46:20,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 16:46:20,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1vmg.c.gcp-upstream.internal','N') ;"
2026-08-12 16:46:21,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:46:21,460 [I] Table x_portal_user does not exist in database ranger
2026-08-12 16:46:21,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:46:21,771 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 16:46:21,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:46:22,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:46:22,394 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 16:46:22,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 16:46:23.704 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.724 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.746 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.765 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.785 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.805 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.871 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.880 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.897 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.909 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.924 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.943 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.956 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.965 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.973 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:23.982 UTC [16065] WARNING: there is no transaction in progress
2026-08-12 16:46:24,887 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 16:46:24,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 16:46:25,162 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 16:46:25,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:46:25,457 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 16:46:25,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:46:25,759 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 16:46:25,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:46:26,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:46:26,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1vmg.c.gcp-upstream.internal';"
2026-08-12 16:46:26,624 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 16:46:26,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 16:46:26,936 [I] DB_PATCHES have already been applied
I20260812 16:46:26.945528 29644 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWMmoHF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 16:46:26.945643 29644 mini_ranger.cc:193] Using host: 127.28.243.20
I20260812 16:46:26.970670 29644 mini_ranger.cc:227] Ranger admin URL: http://127.28.243.20:38013
Aug 12, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 4:46:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 4:46:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 16:46:28.322537 29644 mini_ranger.cc:160] Time spent starting Ranger: real 10.737s user 0.003s sys 0.087s
Aug 12, 2026 4:46:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 4:46:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 4:46:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 4:46:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 4:46:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 16:46:54.535665 29644 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/rangerkms_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/rangerkms_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.243.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 16:46:54 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 16:46:54.810667 29644 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 16:46:54.849085 29644 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 16:46:55.996596 29644 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 16:46:55.996817 29644 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms
I20260812 16:46:55.997430 29644 mini_ranger_kms.cc:227] Using postgres at 127.28.243.20:42975
2026-08-12 16:46:56,699 [I] DB FLAVOR :POSTGRES
2026-08-12 16:46:56,699 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:46:56,699 [I] Checking connection
2026-08-12 16:46:56,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 16:46:56,974 [I] connection success
2026-08-12 16:46:56,975 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 16:46:56,975 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 16:46:56,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 16:46:57,244 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 16:46:57,245 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:46:57,245 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 16:46:57,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:42975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 16:46:57,563 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 16:46:58.182142 29644 mini_ranger_kms.cc:310] Created kms service
I20260812 16:46:58.434894 29644 mini_ranger_kms.cc:326] Created kudu user
I20260812 16:46:58.493366 29644 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 16:46:58 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:46:58.952531 29644 mini_ranger_kms.cc:384] Added ranger policy
I20260812 16:46:58.954186 29644 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 16:46:58.954267 29644 mini_ranger_kms.cc:240] Using host: 127.28.243.20
I20260812 16:46:58.982280 29644 mini_ranger_kms.cc:276] Ranger KMS PID: 16349
I20260812 16:46:58.982498 29644 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.243.20:57155
Aug 12, 2026 4:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 4:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 4:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 4:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 16:47:01 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786553221, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:47:01.721951 29644 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.911s user 0.178s sys 0.567s
I20260812 16:47:01.722368 29644 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 16:47:01.722618 29644 mini_ranger_kms.cc:401] 127.28.243.20:57155/kms/v1/keys
Aug 12 16:47:02 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553221, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.20@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
Aug 12 16:47:02 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553221, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.20@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
Aug 12 16:47:05 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:47:05.158736 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46689
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46689
--ranger_config_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:57155/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:47:05.610146 16432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:47:05.610553 16432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:47:05.610718 16432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:47:05.619947 16432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:47:05.620102 16432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:47:05.620182 16432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:47:05.620261 16432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:47:05.620396 16432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:47:05.634058 16432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.243.20:57155/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46689
--ranger_config_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:47:05.635670 16432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:47:05.637228 16432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:47:05.653139 16439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:05.653976 16442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:05.654023 16438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:47:05.654760 16432 server_base.cc:1034] running on GCE node
I20260812 16:47:05.655652 16432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:47:05.657508 16432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:47:05.658900 16432 hybrid_clock.cc:648] HybridClock initialized: now 1786553225658756 us; error 55 us; skew 500 ppm
I20260812 16:47:05.659420 16432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:47:05 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:47:05.667043 16432 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:47:05.669099 16432 webserver.cc:460] Webserver started at http://127.28.243.62:46831/ using document root <none> and password file <none>
I20260812 16:47:05.670491 16432 fs_manager.cc:359] Metadata directory not provided
I20260812 16:47:05.670801 16432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:47:05.671384 16432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:47:05 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:47:05.730330 16432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "c774ccb91da041d4851fa7e8b31caa51"
format_stamp: "Formatted at 2026-08-12 16:47:05 on dist-test-slave-1vmg"
server_key: "48110c501a7db51924c318f885d399ff"
server_key_iv: "40d1a957f93f4e9b18f47cfe4226d93c"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:05.731526 16432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "c774ccb91da041d4851fa7e8b31caa51"
format_stamp: "Formatted at 2026-08-12 16:47:05 on dist-test-slave-1vmg"
server_key: "48110c501a7db51924c318f885d399ff"
server_key_iv: "40d1a957f93f4e9b18f47cfe4226d93c"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:05.738440 16432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:47:05.761134 16454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:05.763624 16432 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260812 16:47:05.763823 16432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "c774ccb91da041d4851fa7e8b31caa51"
format_stamp: "Formatted at 2026-08-12 16:47:05 on dist-test-slave-1vmg"
server_key: "48110c501a7db51924c318f885d399ff"
server_key_iv: "40d1a957f93f4e9b18f47cfe4226d93c"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:05.764075 16432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:47:05.794835 16432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:47:05.799423 16432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:47:05.799773 16432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:47:05.824658 16432 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:46689
I20260812 16:47:05.824703 16534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:46689 every 8 connection(s)
I20260812 16:47:05.826220 16432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:47:05.830574 16535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:47:05.833056 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 16432
I20260812 16:47:05.833292 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:47:05.834255 29644 external_mini_cluster.cc:1428] Setting key 623b267a30579f330ee932d2aff9b3d5
I20260812 16:47:05.845165 16535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51: Bootstrap starting.
I20260812 16:47:05.849753 16535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51: Neither blocks nor log segments found. Creating new log.
Aug 12 16:47:05 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:47:05.851145 16535 log.cc:824] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51: Log is configured to *not* fsync() on all Append() calls
I20260812 16:47:05.855777 16535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51: No bootstrap required, opened a new log
I20260812 16:47:05.861189 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:05.836780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.28.243.62:46689)
0812 16:47:05.837778 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:05.837795 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:05.837843 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:47:05.839796 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:05.839823 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:05.840255 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:05.840677 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:05.842096 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:05.843617 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:05.845199 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:05.846887 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:05.847138 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:05.851793 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:05.851812 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:05.851840 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:05.851903 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:05.856097 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:05.856959 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:05.856969 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:05.856979 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:05.857182 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:05.857595 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:05.857600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:05.857603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:05.858428 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:05.858753 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:05.859227 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 16:47:05.859712 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":493,"threads_started":1}
I20260812 16:47:05.866135 16535 raft_consensus.cc:348] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c774ccb91da041d4851fa7e8b31caa51" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46689 } }
I20260812 16:47:05.866535 16535 raft_consensus.cc:374] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:47:05.866611 16535 raft_consensus.cc:729] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c774ccb91da041d4851fa7e8b31caa51, State: Initialized, Role: FOLLOWER
I20260812 16:47:05.867216 16535 consensus_queue.cc:260] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c774ccb91da041d4851fa7e8b31caa51" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46689 } }
I20260812 16:47:05.867484 16535 raft_consensus.cc:388] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:47:05.867635 16535 raft_consensus.cc:482] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:47:05.867823 16535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:47:05.870441 16535 raft_consensus.cc:504] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c774ccb91da041d4851fa7e8b31caa51" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46689 } }
I20260812 16:47:05.871171 16535 leader_election.cc:302] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c774ccb91da041d4851fa7e8b31caa51; no voters:
I20260812 16:47:05.871937 16535 leader_election.cc:288] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:47:05.872059 16545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:47:05.872845 16545 raft_consensus.cc:686] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [term 1 LEADER]: Becoming Leader. State: Replica: c774ccb91da041d4851fa7e8b31caa51, State: Running, Role: LEADER
I20260812 16:47:05.873440 16545 consensus_queue.cc:237] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c774ccb91da041d4851fa7e8b31caa51" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46689 } }
I20260812 16:47:05.874287 16535 sys_catalog.cc:564] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:47:05.876215 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c774ccb91da041d4851fa7e8b31caa51. Latest consensus state: current_term: 1 leader_uuid: "c774ccb91da041d4851fa7e8b31caa51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c774ccb91da041d4851fa7e8b31caa51" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46689 } } }
I20260812 16:47:05.876637 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [sys.catalog]: This master's current role is: LEADER
I20260812 16:47:05.878577 16553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:47:05.878495 16547 sys_catalog.cc:455] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c774ccb91da041d4851fa7e8b31caa51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c774ccb91da041d4851fa7e8b31caa51" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46689 } } }
I20260812 16:47:05.878813 16547 sys_catalog.cc:458] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51 [sys.catalog]: This master's current role is: LEADER
I20260812 16:47:05.884970 16553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:47:05.892655 16535 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 16:47:05.907300 16553 catalog_manager.cc:1347] Generated new cluster ID: 4cfecdb5f6024368ae0058b8dbe5b458
I20260812 16:47:05.907586 16553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:47:05.933779 16553 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:47:05.935513 16553 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:47:05.947402 16553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51: Generated new TSK 0
I20260812 16:47:05.948427 16553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:47:06.783427 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:57155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46689
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:47:07 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786553227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 16:47:07.287971 16609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:47:07.288465 16609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:47:07.288686 16609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:47:07.301589 16609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:47:07.301770 16609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:47:07.302021 16609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:47:07.321686 16609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.243.20:57155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:47:07.324044 16609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:47:07.326064 16609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:47:07.368147 16627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:07.369344 16628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:47:07.382611 16609 server_base.cc:1034] running on GCE node
W20260812 16:47:07.383003 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:47:07.384491 16609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:47:07.386991 16609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:47:07.388330 16609 hybrid_clock.cc:648] HybridClock initialized: now 1786553227388217 us; error 107 us; skew 500 ppm
I20260812 16:47:07.388803 16609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:47:07 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:47:07.393646 16609 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:47:07.395984 16609 webserver.cc:460] Webserver started at http://127.28.243.1:37087/ using document root <none> and password file <none>
I20260812 16:47:07.397053 16609 fs_manager.cc:359] Metadata directory not provided
I20260812 16:47:07.397238 16609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:47:07.397642 16609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:47:07 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:47:07.419806 16609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "b31ad6d494b345f1a70a9738b617fb7b"
format_stamp: "Formatted at 2026-08-12 16:47:07 on dist-test-slave-1vmg"
server_key: "e2dd64ac8e430f2c6a7c1cd4378e7900"
server_key_iv: "4c384a288e05b7ae093c5f00e1718d65"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:07.421231 16609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "b31ad6d494b345f1a70a9738b617fb7b"
format_stamp: "Formatted at 2026-08-12 16:47:07 on dist-test-slave-1vmg"
server_key: "e2dd64ac8e430f2c6a7c1cd4378e7900"
server_key_iv: "4c384a288e05b7ae093c5f00e1718d65"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:07.430758 16609 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260812 16:47:07.454386 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:07.456534 16609 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260812 16:47:07.456802 16609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "b31ad6d494b345f1a70a9738b617fb7b"
format_stamp: "Formatted at 2026-08-12 16:47:07 on dist-test-slave-1vmg"
server_key: "e2dd64ac8e430f2c6a7c1cd4378e7900"
server_key_iv: "4c384a288e05b7ae093c5f00e1718d65"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:07.457078 16609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:47:07.531104 16609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:47:07.537797 16609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:47:07.538368 16609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:47:07.539716 16609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:47:07.541489 16609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:47:07.541587 16609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:07.541761 16609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:47:07.541821 16609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:47:07 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:47:07.728415 16838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:35939 every 8 connection(s)
I20260812 16:47:07.728855 16609 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:35939
I20260812 16:47:07.731212 16609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:47:07.741586 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 16609
I20260812 16:47:07.741822 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:47:07.742648 29644 external_mini_cluster.cc:1428] Setting key c8f74e86a4692506405636fe1da4532a
Aug 12 16:47:07 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:47:07.779937 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:07.741359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46689 (local address 127.28.243.1:60413)
0812 16:47:07.748201 (+ 6842us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:07.748282 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:07.750593 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:07.755455 (+ 4862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:07.755471 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:07.756215 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:47:07.757171 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:07.757206 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:07.758967 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:07.758974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:07.761340 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:07.761363 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:07.762728 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:07.762736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:07.763102 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:07.764258 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:47:07.764290 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:47:07.767824 (+ 3534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:47:07.773514 (+ 5690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:47:07.773570 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:47:07.773616 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:47:07.774740 (+ 1124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:47:07.775391 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:47:07.775397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:47:07.775401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:47:07.775784 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:47:07.776663 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:47:07.776674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:47:07.777231 (+ 557us) client_negotiation.cc:769] Sending connection context
0812 16:47:07.777608 (+ 377us) client_negotiation.cc:240] Negotiation successful
0812 16:47:07.778264 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6496,"thread_start_us":6261,"threads_started":1}
I20260812 16:47:07.781401 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:07.742194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:60413 (local address 127.28.243.62:46689)
0812 16:47:07.751441 (+ 9247us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:07.751447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:07.751469 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:47:07.751536 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:07.751541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:07.751630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:07.751853 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:07.757453 (+ 5600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:07.758656 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:07.761577 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:07.762215 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:07.762327 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:07.768202 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:07.768210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:07.768215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:07.768262 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:07.773089 (+ 4827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:07.774992 (+ 1903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:07.774999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:07.775003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:07.775224 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:07.775962 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:07.775967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:07.775970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:07.776411 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:07.776563 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:07.780662 (+ 4099us) server_negotiation.cc:299] Negotiation successful
0812 16:47:07.780984 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9104,"thread_start_us":8939,"threads_started":1}
I20260812 16:47:07.783921 16839 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46689
I20260812 16:47:07.785365 16839 heartbeater.cc:464] Registering TS with master...
I20260812 16:47:07.786365 16839 heartbeater.cc:511] Master 127.28.243.62:46689 requested a full tablet report, sending...
I20260812 16:47:07.789605 16477 ts_manager.cc:194] Registered new tserver with Master: b31ad6d494b345f1a70a9738b617fb7b (127.28.243.1:35939)
I20260812 16:47:07.792579 16477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:60413
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:47:07.846076 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:57155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46689
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:47:08 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
W20260812 16:47:08.398205 16846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:47:08.398535 16846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:47:08.398770 16846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:47:08.407509 16846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:47:08.407670 16846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:47:08.407840 16846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:47:08.420653 16846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.243.20:57155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:47:08.422132 16846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:47:08.423647 16846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:47:08.439440 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:08.439338 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:08.439477 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:47:08.440768 16846 server_base.cc:1034] running on GCE node
I20260812 16:47:08.441962 16846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:47:08.443499 16846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:47:08.444850 16846 hybrid_clock.cc:648] HybridClock initialized: now 1786553228444698 us; error 132 us; skew 500 ppm
I20260812 16:47:08.445363 16846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:47:08 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:47:08.449826 16846 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:47:08.451812 16846 webserver.cc:460] Webserver started at http://127.28.243.2:42541/ using document root <none> and password file <none>
I20260812 16:47:08.452634 16846 fs_manager.cc:359] Metadata directory not provided
I20260812 16:47:08.452765 16846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:47:08.453104 16846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:47:08 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:47:08.469321 16846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "f51471177cb9425e9cfd9b5be80bea0f"
format_stamp: "Formatted at 2026-08-12 16:47:08 on dist-test-slave-1vmg"
server_key: "2cfec3cbd0492666074597965d472d01"
server_key_iv: "9d5022ad81b6b5bc56000edfb37bc59a"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:08.470283 16846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "f51471177cb9425e9cfd9b5be80bea0f"
format_stamp: "Formatted at 2026-08-12 16:47:08 on dist-test-slave-1vmg"
server_key: "2cfec3cbd0492666074597965d472d01"
server_key_iv: "9d5022ad81b6b5bc56000edfb37bc59a"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:08.476711 16846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:47:08.495533 16883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:08.497262 16846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:47:08.497464 16846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "f51471177cb9425e9cfd9b5be80bea0f"
format_stamp: "Formatted at 2026-08-12 16:47:08 on dist-test-slave-1vmg"
server_key: "2cfec3cbd0492666074597965d472d01"
server_key_iv: "9d5022ad81b6b5bc56000edfb37bc59a"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:08.497713 16846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:47:08.541196 16846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:47:08.547220 16846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:47:08.547640 16846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:47:08.549222 16846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:47:08.551133 16846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:47:08.551227 16846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:08.551344 16846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:47:08.551396 16846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:08.592466 16846 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:44739
I20260812 16:47:08.592391 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:44739 every 8 connection(s)
I20260812 16:47:08.593923 16846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:47:08.594487 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 16846
I20260812 16:47:08.594702 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:47:08.595435 29644 external_mini_cluster.cc:1428] Setting key 06d4e9e1fa630c4c2d6fbdbc776d072b
Aug 12 16:47:08 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:47:08.618886 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:08.597905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:56473 (local address 127.28.243.62:46689)
0812 16:47:08.598450 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:08.598455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:08.599937 (+ 1482us) server_negotiation.cc:407] Connection header received
0812 16:47:08.602047 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:08.602052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:08.602160 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:08.602275 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:08.604410 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:08.605309 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:08.607066 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:08.607477 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:08.607598 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:08.611809 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:08.611815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:08.611818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:08.611862 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:08.615033 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:08.616022 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:08.616027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:08.616030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:08.616270 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:08.616741 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:08.616746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:08.616750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:08.617173 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:08.617330 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:08.618229 (+ 899us) server_negotiation.cc:299] Negotiation successful
0812 16:47:08.618499 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
I20260812 16:47:08.620026 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:08.598642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46689 (local address 127.28.243.2:56473)
0812 16:47:08.599728 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:08.599784 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:08.601739 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:08.602600 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:08.602614 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:08.603396 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:47:08.604098 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:08.604120 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:08.605502 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:08.605507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:08.606841 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:08.606884 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:08.607723 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:08.607732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:08.607993 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:08.608737 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:47:08.608767 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:47:08.611502 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:47:08.615232 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:47:08.615242 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:47:08.615260 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:47:08.615865 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:47:08.616394 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:47:08.616398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:47:08.616400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:47:08.616610 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:47:08.617321 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:47:08.617330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:47:08.617718 (+ 388us) client_negotiation.cc:769] Sending connection context
0812 16:47:08.618009 (+ 291us) client_negotiation.cc:240] Negotiation successful
0812 16:47:08.618609 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":460,"threads_started":1}
I20260812 16:47:08.621842 17060 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46689
I20260812 16:47:08.622169 17060 heartbeater.cc:464] Registering TS with master...
I20260812 16:47:08.623063 17060 heartbeater.cc:511] Master 127.28.243.62:46689 requested a full tablet report, sending...
I20260812 16:47:08.625319 16477 ts_manager.cc:194] Registered new tserver with Master: f51471177cb9425e9cfd9b5be80bea0f (127.28.243.2:44739)
I20260812 16:47:08.626583 16477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:56473
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:47:08.668617 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:57155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46689
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:47:08.798542 16839 heartbeater.cc:503] Master 127.28.243.62:46689 was elected leader, sending a full tablet report...
W20260812 16:47:09.094982 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:47:09.095333 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:47:09.095476 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:47:09.103557 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:47:09.103711 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:47:09.103860 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:47:09.116853 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.243.20:57155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:47:09.118396 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:47:09.120010 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:47:09.136754 17075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:09.136710 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:47:09.136718 17074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:47:09.137741 17069 server_base.cc:1034] running on GCE node
I20260812 16:47:09.138417 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:47:09.139598 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:47:09.140838 17069 hybrid_clock.cc:648] HybridClock initialized: now 1786553229140786 us; error 62 us; skew 500 ppm
I20260812 16:47:09.141215 17069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:47:09 dist-test-slave-1vmg krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:47:09.144649 17069 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:47:09.146275 17069 webserver.cc:460] Webserver started at http://127.28.243.3:36823/ using document root <none> and password file <none>
I20260812 16:47:09.147055 17069 fs_manager.cc:359] Metadata directory not provided
I20260812 16:47:09.147194 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:47:09.147545 17069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:47:09 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:47:09.165680 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "d44eff3006c741de820730aeed2d9197"
format_stamp: "Formatted at 2026-08-12 16:47:09 on dist-test-slave-1vmg"
server_key: "4bdb5dc583986a660f50e4deb7bd0221"
server_key_iv: "74232402525f8639cf81a35249553fdc"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:09.166782 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "d44eff3006c741de820730aeed2d9197"
format_stamp: "Formatted at 2026-08-12 16:47:09 on dist-test-slave-1vmg"
server_key: "4bdb5dc583986a660f50e4deb7bd0221"
server_key_iv: "74232402525f8639cf81a35249553fdc"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:09.173516 17069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:47:09.198167 17088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:09.200356 17069 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 16:47:09.200601 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "d44eff3006c741de820730aeed2d9197"
format_stamp: "Formatted at 2026-08-12 16:47:09 on dist-test-slave-1vmg"
server_key: "4bdb5dc583986a660f50e4deb7bd0221"
server_key_iv: "74232402525f8639cf81a35249553fdc"
server_key_version: "kuduclusterkey@0"
I20260812 16:47:09.200902 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:47:09.222710 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:47:09.226830 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:47:09.227144 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:47:09.228175 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:47:09.229701 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:47:09.229785 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:09.229875 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:47:09.229923 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:47:09.273489 17069 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:44125
I20260812 16:47:09.273522 17273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:44125 every 8 connection(s)
I20260812 16:47:09.275513 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:47:09.276364 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 17069
I20260812 16:47:09.276608 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:47:09.277518 29644 external_mini_cluster.cc:1428] Setting key 61f177efa9b2404c257acef49d97280b
Aug 12 16:47:09 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:47:09.303877 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.279192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:57811 (local address 127.28.243.62:46689)
0812 16:47:09.279926 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.279932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.281936 (+ 2004us) server_negotiation.cc:407] Connection header received
0812 16:47:09.284688 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.284693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.284778 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.284880 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:09.286592 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.287416 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.289910 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.290468 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.290596 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.296255 (+ 5659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:09.296261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:09.296265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:09.296319 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:09.299744 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.301135 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.301139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.301142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.301310 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.301895 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.301899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.301902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.302319 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:09.302448 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.303387 (+ 939us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.303597 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":1303,"threads_started":1}
I20260812 16:47:09.304929 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.280221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46689 (local address 127.28.243.3:57811)
0812 16:47:09.281598 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.281785 (+ 187us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:09.284327 (+ 2542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:09.285129 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:09.285139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:09.285688 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:47:09.286385 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.286406 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.287615 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.287620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.289655 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.289674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.291002 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.291017 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.291352 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.292430 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:47:09.292465 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:47:09.295922 (+ 3457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:47:09.300077 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:47:09.300091 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:47:09.300115 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:47:09.300973 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:47:09.301489 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:47:09.301495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:47:09.301499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:47:09.301750 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:47:09.302481 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:47:09.302492 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:47:09.302950 (+ 458us) client_negotiation.cc:769] Sending connection context
0812 16:47:09.303277 (+ 327us) client_negotiation.cc:240] Negotiation successful
0812 16:47:09.303764 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":862,"threads_started":1}
I20260812 16:47:09.307180 17275 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46689
I20260812 16:47:09.307559 17275 heartbeater.cc:464] Registering TS with master...
I20260812 16:47:09.309022 17275 heartbeater.cc:511] Master 127.28.243.62:46689 requested a full tablet report, sending...
I20260812 16:47:09.311004 16477 ts_manager.cc:194] Registered new tserver with Master: d44eff3006c741de820730aeed2d9197 (127.28.243.3:44125)
I20260812 16:47:09.311913 16477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:57811
I20260812 16:47:09.321978 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:47:09.346725 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.328495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.28.243.62:46689)
0812 16:47:09.328758 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.328767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.329906 (+ 1139us) server_negotiation.cc:407] Connection header received
0812 16:47:09.330097 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.330103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.330213 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.330330 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:09.332400 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.333233 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.335125 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.335742 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.335867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.339497 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:09.339504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:09.339508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:09.339557 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:09.342536 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.343872 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.343877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.343879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.344085 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.344742 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.344749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.344752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.345350 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:09.345660 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.346224 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.346464 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:47:09.351662 16477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:47:09.359854 16477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:47:09.392005 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.376530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.28.243.1:35939)
0812 16:47:09.377698 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.377705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.378169 (+ 464us) server_negotiation.cc:407] Connection header received
0812 16:47:09.378370 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.378380 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.378779 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.381815 (+ 3036us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:47:09.382261 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.384822 (+ 2561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.389134 (+ 4312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.390655 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.390838 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.391115 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.391424 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.391627 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":781,"threads_started":1}
I20260812 16:47:09.392103 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.380090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.28.243.2:44739)
0812 16:47:09.383147 (+ 3057us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.383154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.383182 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:47:09.383290 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.383299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.383530 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.383737 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:47:09.384432 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.386219 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.389127 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.390340 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.390494 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.390664 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.391562 (+ 898us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.391771 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2880,"thread_start_us":2732,"threads_started":1}
I20260812 16:47:09.392194 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.375477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53696)
0812 16:47:09.377627 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.377688 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:09.377991 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:09.379319 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:09.379330 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:09.379376 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:47:09.381565 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.381583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.387098 (+ 5515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.387102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.388874 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.388891 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.390908 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.390941 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.391210 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.391244 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:47:09.391345 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 16:47:09.391593 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1849,"mutex_wait_us":63,"thread_start_us":883,"threads_started":1}
I20260812 16:47:09.392855 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.376331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:44125 (local address 127.0.0.1:60720)
0812 16:47:09.378666 (+ 2335us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.378685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:09.378813 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:09.381103 (+ 2290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:09.381108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:09.381135 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:47:09.382045 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.382061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.384989 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.384994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.388359 (+ 3365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.388378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.391741 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.391748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.392020 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.392054 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:47:09.392134 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 16:47:09.392299 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2209,"mutex_wait_us":2,"thread_start_us":2041,"threads_started":1}
I20260812 16:47:09.392886 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.379695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:44739 (local address 127.0.0.1:56824)
0812 16:47:09.380824 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.380854 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:09.381013 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:09.383772 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:09.383777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:09.383807 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:47:09.384244 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.384255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.386611 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.386618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.388940 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.388952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.390821 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.390828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.391112 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.391197 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 16:47:09.391345 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 16:47:09.392519 (+ 1174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":1030,"thread_start_us":600,"threads_started":1}
I20260812 16:47:09.393851 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.376907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.28.243.3:44125)
0812 16:47:09.379922 (+ 3015us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.379931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.379971 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:47:09.380092 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.380101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.380412 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.380660 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:47:09.382285 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.384807 (+ 2522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.388577 (+ 3770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.391470 (+ 2893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.393009 (+ 1539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.393218 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.393322 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.393479 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2794,"thread_start_us":2584,"threads_started":1}
I20260812 16:47:09.397125 16969 tablet_service.cc:1467] Processing CreateTablet for tablet 80ec25cf208144ac96f3671d39844645 (DEFAULT_TABLE table=test-table [id=81531e19abc445e8927e8016013bee41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:47:09.397511 16738 tablet_service.cc:1467] Processing CreateTablet for tablet 80ec25cf208144ac96f3671d39844645 (DEFAULT_TABLE table=test-table [id=81531e19abc445e8927e8016013bee41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:47:09.398753 16969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ec25cf208144ac96f3671d39844645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:47:09.398444 17168 tablet_service.cc:1467] Processing CreateTablet for tablet 80ec25cf208144ac96f3671d39844645 (DEFAULT_TABLE table=test-table [id=81531e19abc445e8927e8016013bee41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:47:09.399514 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ec25cf208144ac96f3671d39844645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:47:09.400746 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ec25cf208144ac96f3671d39844645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:47:09.416327 17314 tablet_bootstrap.cc:492] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Bootstrap starting.
I20260812 16:47:09.420617 17316 tablet_bootstrap.cc:492] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Bootstrap starting.
I20260812 16:47:09.423025 17314 tablet_bootstrap.cc:654] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Neither blocks nor log segments found. Creating new log.
I20260812 16:47:09.423182 17320 tablet_bootstrap.cc:492] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Bootstrap starting.
I20260812 16:47:09.425074 17314 log.cc:824] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:47:09.426829 17316 tablet_bootstrap.cc:654] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Neither blocks nor log segments found. Creating new log.
I20260812 16:47:09.428442 17316 log.cc:824] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:47:09.429000 17320 tablet_bootstrap.cc:654] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Neither blocks nor log segments found. Creating new log.
I20260812 16:47:09.430579 17314 tablet_bootstrap.cc:492] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: No bootstrap required, opened a new log
I20260812 16:47:09.431111 17314 ts_tablet_manager.cc:1397] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260812 16:47:09.431308 17320 log.cc:824] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Log is configured to *not* fsync() on all Append() calls
I20260812 16:47:09.432073 17316 tablet_bootstrap.cc:492] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: No bootstrap required, opened a new log
I20260812 16:47:09.432587 17316 ts_tablet_manager.cc:1397] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260812 16:47:09.436272 17320 tablet_bootstrap.cc:492] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: No bootstrap required, opened a new log
I20260812 16:47:09.436681 17320 ts_tablet_manager.cc:1397] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260812 16:47:09.443131 17316 raft_consensus.cc:348] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.443678 17316 raft_consensus.cc:374] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:47:09.443812 17316 raft_consensus.cc:729] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ad6d494b345f1a70a9738b617fb7b, State: Initialized, Role: FOLLOWER
I20260812 16:47:09.444535 17316 consensus_queue.cc:260] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.444798 17314 raft_consensus.cc:348] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.445393 17314 raft_consensus.cc:374] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:47:09.445546 17314 raft_consensus.cc:729] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51471177cb9425e9cfd9b5be80bea0f, State: Initialized, Role: FOLLOWER
I20260812 16:47:09.446321 17314 consensus_queue.cc:260] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.448412 17060 heartbeater.cc:503] Master 127.28.243.62:46689 was elected leader, sending a full tablet report...
I20260812 16:47:09.448563 17320 raft_consensus.cc:348] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.449399 17320 raft_consensus.cc:374] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:47:09.449515 17314 ts_tablet_manager.cc:1428] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260812 16:47:09.450117 17320 raft_consensus.cc:729] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44eff3006c741de820730aeed2d9197, State: Initialized, Role: FOLLOWER
I20260812 16:47:09.450842 17316 ts_tablet_manager.cc:1428] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260812 16:47:09.451289 17320 consensus_queue.cc:260] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.452768 17275 heartbeater.cc:503] Master 127.28.243.62:46689 was elected leader, sending a full tablet report...
I20260812 16:47:09.453719 17320 ts_tablet_manager.cc:1428] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Time spent starting tablet: real 0.017s user 0.006s sys 0.011s
W20260812 16:47:09.499380 16840 tablet.cc:2369] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:47:09.529884 17276 tablet.cc:2369] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:47:09.545674 17331 raft_consensus.cc:482] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:47:09.546010 17331 raft_consensus.cc:504] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.547921 17331 leader_election.cc:288] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b31ad6d494b345f1a70a9738b617fb7b (127.28.243.1:35939), f51471177cb9425e9cfd9b5be80bea0f (127.28.243.2:44739)
I20260812 16:47:09.555811 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.548279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:44739 (local address 127.28.243.3:37009)
0812 16:47:09.548577 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.548599 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:09.548765 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:09.549508 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:09.549516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:09.549544 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:47:09.549733 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.549742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.551253 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.551259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.553214 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.553233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.554972 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.554980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.555185 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.555223 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:47:09.555341 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 16:47:09.555496 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260812 16:47:09.556095 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.548541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:37009 (local address 127.28.243.2:44739)
0812 16:47:09.548753 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.548759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.548785 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:47:09.548906 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.548911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.549232 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.549450 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:47:09.550025 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.551099 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.553448 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.554751 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.554889 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.555081 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.555494 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.555748 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:47:09.557030 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.548586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:34737 (local address 127.28.243.1:35939)
0812 16:47:09.548775 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.548780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.549931 (+ 1151us) server_negotiation.cc:407] Connection header received
0812 16:47:09.550288 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.550292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.550356 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.550436 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:47:09.551096 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.552219 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.554547 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.555730 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.555941 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.556027 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.556623 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.556775 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:47:09.557120 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.548360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.28.243.3:34737)
0812 16:47:09.549779 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.549802 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:47:09.550078 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:47:09.550480 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:47:09.550485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:47:09.550513 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:47:09.550877 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.550889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.552462 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.552466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.554298 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:47:09.554316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.556211 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.556218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:47:09.556392 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.556423 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 16:47:09.556558 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 16:47:09.556702 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":1,"thread_start_us":812,"threads_started":1}
I20260812 16:47:09.557916 16997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec25cf208144ac96f3671d39844645" candidate_uuid: "d44eff3006c741de820730aeed2d9197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f51471177cb9425e9cfd9b5be80bea0f" is_pre_election: true
I20260812 16:47:09.558509 16997 raft_consensus.cc:2454] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44eff3006c741de820730aeed2d9197 in term 0.
I20260812 16:47:09.558617 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec25cf208144ac96f3671d39844645" candidate_uuid: "d44eff3006c741de820730aeed2d9197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ad6d494b345f1a70a9738b617fb7b" is_pre_election: true
I20260812 16:47:09.559369 16767 raft_consensus.cc:2454] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44eff3006c741de820730aeed2d9197 in term 0.
I20260812 16:47:09.559423 17094 leader_election.cc:302] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d44eff3006c741de820730aeed2d9197, f51471177cb9425e9cfd9b5be80bea0f; no voters:
I20260812 16:47:09.559880 17331 raft_consensus.cc:2781] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:47:09.560045 17331 raft_consensus.cc:482] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:47:09.560153 17331 raft_consensus.cc:3037] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:47:09.563127 17331 raft_consensus.cc:504] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.563896 17331 leader_election.cc:288] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [CANDIDATE]: Term 1 election: Requested vote from peers b31ad6d494b345f1a70a9738b617fb7b (127.28.243.1:35939), f51471177cb9425e9cfd9b5be80bea0f (127.28.243.2:44739)
I20260812 16:47:09.564589 16997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec25cf208144ac96f3671d39844645" candidate_uuid: "d44eff3006c741de820730aeed2d9197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f51471177cb9425e9cfd9b5be80bea0f"
I20260812 16:47:09.564620 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec25cf208144ac96f3671d39844645" candidate_uuid: "d44eff3006c741de820730aeed2d9197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ad6d494b345f1a70a9738b617fb7b"
I20260812 16:47:09.564812 16997 raft_consensus.cc:3037] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:47:09.564944 16767 raft_consensus.cc:3037] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:47:09.567881 16767 raft_consensus.cc:2454] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44eff3006c741de820730aeed2d9197 in term 1.
I20260812 16:47:09.568101 16997 raft_consensus.cc:2454] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44eff3006c741de820730aeed2d9197 in term 1.
I20260812 16:47:09.568867 17091 leader_election.cc:302] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b31ad6d494b345f1a70a9738b617fb7b, d44eff3006c741de820730aeed2d9197; no voters:
I20260812 16:47:09.569412 17331 raft_consensus.cc:2781] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:47:09.570063 17331 raft_consensus.cc:686] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 1 LEADER]: Becoming Leader. State: Replica: d44eff3006c741de820730aeed2d9197, State: Running, Role: LEADER
I20260812 16:47:09.570864 17331 consensus_queue.cc:237] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } }
I20260812 16:47:09.577064 16475 catalog_manager.cc:5495] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 reported cstate change: term changed from 0 to 1, leader changed from <none> to d44eff3006c741de820730aeed2d9197 (127.28.243.3). New cstate: current_term: 1 leader_uuid: "d44eff3006c741de820730aeed2d9197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d44eff3006c741de820730aeed2d9197" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 44125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 } health_report { overall_health: UNKNOWN } } }
W20260812 16:47:09.596864 17061 tablet.cc:2369] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:47:09.640129 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.632149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.28.243.3:44125)
0812 16:47:09.632397 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.632402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.632422 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:47:09.632564 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.632567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.632716 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.632858 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:47:09.633461 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.634375 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.637352 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.637801 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.637940 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.638578 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:47:09.639397 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:47:09.639580 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.639693 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.639823 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 16:47:09.650280 16767 raft_consensus.cc:1261] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 1 FOLLOWER]: Refusing update from remote peer d44eff3006c741de820730aeed2d9197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:47:09.650347 16997 raft_consensus.cc:1261] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 1 FOLLOWER]: Refusing update from remote peer d44eff3006c741de820730aeed2d9197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:47:09.651631 17338 consensus_queue.cc:1035] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f51471177cb9425e9cfd9b5be80bea0f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:47:09.652055 17331 consensus_queue.cc:1035] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31ad6d494b345f1a70a9738b617fb7b" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:47:09.667963 17341 mvcc.cc:205] Tried to move back new op lower bound from 7317722028633473024 to 7317722028327022592. Current Snapshot: MvccSnapshot[applied={T|T < 7317722028633473024}]
I20260812 16:47:09.685707 16477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:47:09.686007 16477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:47:09.691021 16477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c774ccb91da041d4851fa7e8b31caa51: Sending DeleteTablet for 3 replicas of tablet 80ec25cf208144ac96f3671d39844645
I20260812 16:47:09.692539 17168 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ec25cf208144ac96f3671d39844645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:47:09 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60720
I20260812 16:47:09.692529 16738 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ec25cf208144ac96f3671d39844645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:47:09 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:53696
I20260812 16:47:09.693966 17366 tablet_replica.cc:330] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: stopping tablet replica
I20260812 16:47:09.693872 16969 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ec25cf208144ac96f3671d39844645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:47:09 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:56824
I20260812 16:47:09.694341 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 16609
I20260812 16:47:09.694506 17366 raft_consensus.cc:2229] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:47:09.695278 17366 raft_consensus.cc:2258] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:47:09.699124 17366 ts_tablet_manager.cc:1905] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:47:09.706868 17370 tablet_replica.cc:330] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: stopping tablet replica
I20260812 16:47:09.706856 17364 tablet_replica.cc:330] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: stopping tablet replica
I20260812 16:47:09.707476 17370 raft_consensus.cc:2229] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:47:09.707625 17364 raft_consensus.cc:2229] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:47:09.708146 17370 raft_consensus.cc:2258] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:47:09.708330 17364 raft_consensus.cc:2258] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 16:47:09 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:47:09.709949 17366 ts_tablet_manager.cc:1918] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:47:09.710163 17366 log.cc:1196] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/80ec25cf208144ac96f3671d39844645
I20260812 16:47:09.710780 17366 ts_tablet_manager.cc:1939] T 80ec25cf208144ac96f3671d39844645 P d44eff3006c741de820730aeed2d9197: Deleting consensus metadata
I20260812 16:47:09.711354 17364 ts_tablet_manager.cc:1905] T 80ec25cf208144ac96f3671d39844645 P b31ad6d494b345f1a70a9738b617fb7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:47:09.713052 16460 catalog_manager.cc:4841] TS d44eff3006c741de820730aeed2d9197 (127.28.243.3:44125): tablet 80ec25cf208144ac96f3671d39844645 (table test-table [id=81531e19abc445e8927e8016013bee41]) successfully deleted
I20260812 16:47:09.714893 17370 ts_tablet_manager.cc:1905] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:47:09.724220 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.695398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.28.243.1:35939)
0812 16:47:09.695598 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.695605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.696440 (+ 835us) server_negotiation.cc:407] Connection header received
0812 16:47:09.697063 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.697070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.697159 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.697240 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:09.699604 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.700839 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.703255 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.703940 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.704057 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.710447 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:09.710467 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:09.710487 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:09.710546 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:09.716522 (+ 5976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.718213 (+ 1691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.718226 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.718230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.718491 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.719946 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.720049 (+ 103us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.720054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.721160 (+ 1106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:09.721522 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.723480 (+ 1958us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.723798 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:47:09.725169 16831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:47:09.725868 17370 ts_tablet_manager.cc:1918] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:47:09.726089 17370 log.cc:1196] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/80ec25cf208144ac96f3671d39844645
I20260812 16:47:09.726646 17370 ts_tablet_manager.cc:1939] T 80ec25cf208144ac96f3671d39844645 P f51471177cb9425e9cfd9b5be80bea0f: Deleting consensus metadata
I20260812 16:47:09.729017 16458 catalog_manager.cc:4841] TS f51471177cb9425e9cfd9b5be80bea0f (127.28.243.2:44739): tablet 80ec25cf208144ac96f3671d39844645 (table test-table [id=81531e19abc445e8927e8016013bee41]) successfully deleted
I20260812 16:47:09.859370 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 16609
I20260812 16:47:09.896894 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 16846
W20260812 16:47:09.896862 16455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:35939 (error 108)
I20260812 16:47:09.899253 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.898468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53712)
0812 16:47:09.898909 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.899056 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":240,"threads_started":1}
W20260812 16:47:09.899537 16455 catalog_manager.cc:4584] TS b31ad6d494b345f1a70a9738b617fb7b (127.28.243.1:35939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80ec25cf208144ac96f3671d39844645: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Aug 12 16:47:09 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:47:09.917316 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.898157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.28.243.2:44739)
0812 16:47:09.898380 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:09.898387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:09.898410 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:47:09.898482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:09.898488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:09.898577 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:09.898692 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:09.900805 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.901731 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.903832 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:09.904387 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:09.904643 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:09.910314 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:09.910335 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:09.910352 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:09.910418 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:09.913527 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.914445 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.914453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.914456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.914619 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:09.915223 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:09.915227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:09.915230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:09.915687 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:09.915997 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:09.916356 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 16:47:09.916607 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 16:47:09.918330 17052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:47:09.953634 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.953055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53722)
0812 16:47:09.953321 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.953453 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 16:47:09.955013 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:09.954502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53726)
0812 16:47:09.954703 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:09.954806 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 16:47:10.014559 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 16846
I20260812 16:47:10.019647 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:10.019012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53738)
0812 16:47:10.019328 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:10.019444 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260812 16:47:10.021207 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:10.020634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53754)
0812 16:47:10.020887 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:10.020992 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260812 16:47:10.056620 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 17069
Aug 12 16:47:10 dist-test-slave-1vmg krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:47:10.073763 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:10.057520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.28.243.3:44125)
0812 16:47:10.057782 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 16:47:10.057788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:47:10.057812 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:47:10.058017 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:47:10.058022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:47:10.058107 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:47:10.058196 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:47:10.059651 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:10.060416 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:10.061683 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:47:10.062177 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:47:10.062301 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:47:10.067086 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:47:10.067107 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:47:10.067127 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:47:10.067175 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:47:10.070434 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:10.071336 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:10.071344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:10.071346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:10.071530 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:47:10.071957 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:47:10.071961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:47:10.071963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:47:10.072411 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:47:10.072739 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 16:47:10.073098 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 16:47:10.073365 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260812 16:47:10.074519 17267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:47:10.102160 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:10.101677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53756)
0812 16:47:10.101914 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:10.102013 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 16:47:10.103487 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:10.103065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53770)
0812 16:47:10.103256 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:10.103336 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 16:47:10.217551 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 17069
I20260812 16:47:10.247570 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 16432
I20260812 16:47:10.248869 16525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:47:10.368914 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:47:10.368330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35939 (local address 127.0.0.1:53784)
0812 16:47:10.368606 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 16:47:10.368727 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 16:47:10.369467 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 16432
I20260812 16:47:10.408692 29644 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 16:47:10.757486 29644 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 16:47:10.758010 29644 mini_ranger.cc:66] Stopping Ranger...
I20260812 16:47:11.116284 29644 mini_ranger.cc:68] Stopped Ranger
2026-08-12 16:47:11.116 UTC [15762] LOG: received smart shutdown request
2026-08-12 16:47:11.121 UTC [15762] LOG: background worker "logical replication launcher" (PID 15773) exited with exit code 1
2026-08-12 16:47:11.122 UTC [15766] LOG: shutting down
2026-08-12 16:47:11.336 UTC [15762] LOG: database system is shut down
2026-08-12T16:47:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (60314 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:47:11 dist-test-slave-1vmg krb5kdc[17391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:47:11 dist-test-slave-1vmg krb5kdc[17391](info): set up 2 sockets
Aug 12 16:47:11 dist-test-slave-1vmg krb5kdc[17391](info): commencing operation
krb5kdc: starting...
W20260812 16:47:12.837457 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:47:13 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:47:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/rangeradmin_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/rangeradmin_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/rangerlookup_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/rangerlookup_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
I20260812 16:47:13.343580 29644 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWMmoHF/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/postgres -l logfile start
2026-08-12 16:47:17.372 UTC [17437] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 16:47:17.372 UTC [17437] LOG: listening on IPv4 address "127.28.243.20", port 54815
2026-08-12 16:47:17.378 UTC [17437] LOG: listening on Unix socket "/tmp/.s.PGSQL.54815"
2026-08-12 16:47:17.409 UTC [17439] LOG: database system was shut down at 2026-08-12 16:47:15 UTC
2026-08-12 16:47:17.415 UTC [17437] LOG: database system is ready to accept connections
I20260812 16:47:18.638090 29644 mini_postgres.cc:96] Postgres bound to 54815
2026-08-12 16:47:18.667 UTC [17455] FATAL: database "slave" does not exist
127.28.243.20:54815 - accepting connections
I20260812 16:47:18.668463 29644 mini_ranger.cc:161] Starting Ranger...
I20260812 16:47:18.710057 29644 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 16:47:19.386065 29644 mini_ranger.cc:87] Created ranger Postgres database
I20260812 16:47:19.386222 29644 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-admin
2026-08-12 16:47:19,753 [I] DB FLAVOR :POSTGRES
2026-08-12 16:47:19,754 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:47:19,754 [I] Checking connection..
2026-08-12 16:47:19,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 16:47:20,097 [I] Checking connection passed.
2026-08-12 16:47:20,097 [I] --------- Verifying version history table ---------
2026-08-12 16:47:20,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:47:20,434 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:47:20,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:47:20,773 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:47:20,773 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 16:47:20,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 16:47:21.097 UTC [17582] WARNING: there is no transaction in progress
2026-08-12 16:47:21,106 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 16:47:21,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:47:21,446 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 16:47:21,446 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:47:21,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 16:47:21,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 16:47:22,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1vmg.c.gcp-upstream.internal','N') ;"
2026-08-12 16:47:22,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:47:22,672 [I] Table x_portal_user does not exist in database ranger
2026-08-12 16:47:22,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:47:23,016 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 16:47:23,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:47:23,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:47:23,623 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 16:47:23,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 16:47:24.982 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.005 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.025 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.045 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.072 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.097 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.177 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.189 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.209 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.225 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.240 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.260 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.273 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.283 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.293 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:25.303 UTC [17778] WARNING: there is no transaction in progress
2026-08-12 16:47:26,315 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 16:47:26,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 16:47:26,610 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 16:47:26,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:47:26,947 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 16:47:26,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:47:27,286 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 16:47:27,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:47:27,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:47:27,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1vmg.c.gcp-upstream.internal';"
2026-08-12 16:47:28,282 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 16:47:28,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 16:47:28,613 [I] DB_PATCHES have already been applied
I20260812 16:47:28.621778 29644 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWMmoHF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 16:47:28.621948 29644 mini_ranger.cc:193] Using host: 127.28.243.20
I20260812 16:47:28.652190 29644 mini_ranger.cc:227] Ranger admin URL: http://127.28.243.20:37615
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 4:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 16:47:30.084127 29644 mini_ranger.cc:160] Time spent starting Ranger: real 11.416s user 0.000s sys 0.091s
Aug 12, 2026 4:47:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 4:47:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 4:47:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 4:47:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 4:47:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 16:47:55.046430 29644 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.243.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.243.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/rangerkms_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.243.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/rangerkms_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.243.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/HTTP_127.28.243.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 16:47:55 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553275, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 16:47:55.297951 29644 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 16:47:55.340088 29644 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 16:47:56.443327 29644 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 16:47:56.443519 29644 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms
I20260812 16:47:56.443570 29644 mini_ranger_kms.cc:227] Using postgres at 127.28.243.20:54815
2026-08-12 16:47:56,896 [I] DB FLAVOR :POSTGRES
2026-08-12 16:47:56,897 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:47:56,897 [I] Checking connection
2026-08-12 16:47:56,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 16:47:57,194 [I] connection success
2026-08-12 16:47:57,194 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 16:47:57,194 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 16:47:57,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 16:47:57,493 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 16:47:57,494 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:47:57,494 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 16:47:57,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.20:54815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 16:47:57,817 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 16:47:58.433481 29644 mini_ranger_kms.cc:310] Created kms service
I20260812 16:47:58.678995 29644 mini_ranger_kms.cc:326] Created kudu user
I20260812 16:47:58.741214 29644 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 16:47:58 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553275, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:47:59.243574 29644 mini_ranger_kms.cc:384] Added ranger policy
I20260812 16:47:59.243938 29644 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWMmoHF/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWMmoHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 16:47:59.244004 29644 mini_ranger_kms.cc:240] Using host: 127.28.243.20
I20260812 16:47:59.272394 29644 mini_ranger_kms.cc:276] Ranger KMS PID: 18096
I20260812 16:47:59.272572 29644 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.243.20:33957
Aug 12, 2026 4:48:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:48:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:48:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 4:48:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 4:48:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 4:48:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 16:48:01 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786553281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:02.078892 29644 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.781s user 0.118s sys 0.382s
I20260812 16:48:02.079211 29644 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 16:48:02.079387 29644 mini_ranger_kms.cc:401] 127.28.243.20:33957/kms/v1/keys
Aug 12 16:48:02 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.20@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
Aug 12 16:48:02 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.20@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
Aug 12 16:48:05 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:48:05.437237 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36643
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:36643
--ranger_config_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:05.846359 18182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:05.846690 18182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:05.846805 18182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:05.855294 18182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:05.855444 18182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:05.855511 18182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:05.855594 18182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:05.855659 18182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:05.866945 18182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.243.20:33957/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:36643
--ranger_config_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:05.869454 18182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:05.871654 18182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:05.887398 18190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:05.888278 18188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:05.887372 18192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:05.889139 18182 server_base.cc:1034] running on GCE node
I20260812 16:48:05.890579 18182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:05.893188 18182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:05.894515 18182 hybrid_clock.cc:648] HybridClock initialized: now 1786553285894464 us; error 52 us; skew 500 ppm
I20260812 16:48:05.895071 18182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:05 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:05.905874 18182 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:05.908713 18182 webserver.cc:460] Webserver started at http://127.28.243.62:33667/ using document root <none> and password file <none>
I20260812 16:48:05.909559 18182 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:05.909684 18182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:05.910048 18182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:48:05 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:05.983987 18182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "fda91b48b0e44fbfa46df437bb1b30a9"
format_stamp: "Formatted at 2026-08-12 16:48:05 on dist-test-slave-1vmg"
server_key: "8c1c82e481ea549c0f8e844d1b0456d6"
server_key_iv: "b7f097e0177dbc879047d60f89a4227f"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:05.985002 18182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "fda91b48b0e44fbfa46df437bb1b30a9"
format_stamp: "Formatted at 2026-08-12 16:48:05 on dist-test-slave-1vmg"
server_key: "8c1c82e481ea549c0f8e844d1b0456d6"
server_key_iv: "b7f097e0177dbc879047d60f89a4227f"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:05.991505 18182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:48:06.014204 18203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:06.015868 18182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:48:06.016078 18182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "fda91b48b0e44fbfa46df437bb1b30a9"
format_stamp: "Formatted at 2026-08-12 16:48:05 on dist-test-slave-1vmg"
server_key: "8c1c82e481ea549c0f8e844d1b0456d6"
server_key_iv: "b7f097e0177dbc879047d60f89a4227f"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:06.016325 18182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:06.034587 18182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:06.039383 18182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:06.039796 18182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:06.063737 18182 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:36643
I20260812 16:48:06.065342 18182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:06.063745 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:36643 every 8 connection(s)
I20260812 16:48:06.070499 18317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:06.074084 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 18182
I20260812 16:48:06.074321 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:48:06.075292 29644 external_mini_cluster.cc:1428] Setting key a636a8ceabc07eb625a4ae67312e7cfc
I20260812 16:48:06.087177 18317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Bootstrap starting.
I20260812 16:48:06.091899 18317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:06.093370 18317 log.cc:824] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:06.099206 18317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: No bootstrap required, opened a new log
Aug 12 16:48:06 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:06.111292 18317 raft_consensus.cc:348] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:06.111927 18317 raft_consensus.cc:374] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:06.112126 18317 raft_consensus.cc:729] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda91b48b0e44fbfa46df437bb1b30a9, State: Initialized, Role: FOLLOWER
I20260812 16:48:06.112920 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:06.078553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.28.243.62:36643)
0812 16:48:06.088891 (+ 10338us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:06.088911 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:06.089399 (+ 488us) server_negotiation.cc:407] Connection header received
0812 16:48:06.091663 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:06.091693 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:06.092230 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:06.092663 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:06.094350 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:06.096331 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:06.097854 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:06.098583 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:06.098772 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:06.102959 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:06.102977 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:06.103004 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:06.103056 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:06.106462 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:06.107440 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:06.107449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:06.107458 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:06.107660 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:06.108099 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:06.108103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:06.108105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:06.109266 (+ 1161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:06.109783 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:06.110533 (+ 750us) server_negotiation.cc:299] Negotiation successful
0812 16:48:06.111225 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10041,"thread_start_us":9744,"threads_started":1}
I20260812 16:48:06.112990 18317 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:06.113350 18317 raft_consensus.cc:388] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:06.113509 18317 raft_consensus.cc:482] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:06.113752 18317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:06.117259 18317 raft_consensus.cc:504] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:06.117950 18317 leader_election.cc:302] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda91b48b0e44fbfa46df437bb1b30a9; no voters:
I20260812 16:48:06.118707 18317 leader_election.cc:288] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:48:06.118837 18326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:06.120282 18326 raft_consensus.cc:686] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 LEADER]: Becoming Leader. State: Replica: fda91b48b0e44fbfa46df437bb1b30a9, State: Running, Role: LEADER
I20260812 16:48:06.121400 18317 sys_catalog.cc:564] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:06.121974 18326 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:06.125782 18329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } } }
I20260812 16:48:06.126250 18329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:06.126807 18332 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda91b48b0e44fbfa46df437bb1b30a9. Latest consensus state: current_term: 1 leader_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } } }
I20260812 16:48:06.127182 18332 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:06.136821 18340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:06.143203 18340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:06.170069 18340 catalog_manager.cc:1347] Generated new cluster ID: 5792fe0ec5104c30889b5441d1e7e9c9
I20260812 16:48:06.170213 18340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:06.220091 18340 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:48:06.221771 18340 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:06.226887 18317 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 16:48:06.235005 18340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Generated new TSK 0
I20260812 16:48:06.236338 18340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:48:07.152132 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36643
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:48:07 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786553287, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 16:48:07.657572 18400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:07.658051 18400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:07.658295 18400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:07.668188 18400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:07.668354 18400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:07.668517 18400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:48:07.680917 18400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.243.20:33957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:07.682480 18400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:07.683976 18400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:07.718086 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:07.723433 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:07.729821 18400 server_base.cc:1034] running on GCE node
W20260812 16:48:07.729661 18417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:07.731333 18400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:07.739398 18400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:07.743011 18400 hybrid_clock.cc:648] HybridClock initialized: now 1786553287742907 us; error 109 us; skew 500 ppm
I20260812 16:48:07.743527 18400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:07 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553287, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:07.748793 18400 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:48:07.752562 18400 webserver.cc:460] Webserver started at http://127.28.243.1:46563/ using document root <none> and password file <none>
I20260812 16:48:07.753695 18400 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:07.753933 18400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:07.754312 18400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:48:07 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553287, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:07.780023 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "5446009ccf1544e6a244064075e68b63"
format_stamp: "Formatted at 2026-08-12 16:48:07 on dist-test-slave-1vmg"
server_key: "e56f0e69058297281729e1ffea214182"
server_key_iv: "55af6f7ffb9fa57f63d040740357476c"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:07.781425 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "5446009ccf1544e6a244064075e68b63"
format_stamp: "Formatted at 2026-08-12 16:48:07 on dist-test-slave-1vmg"
server_key: "e56f0e69058297281729e1ffea214182"
server_key_iv: "55af6f7ffb9fa57f63d040740357476c"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:07.790719 18400 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260812 16:48:07.827708 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:07.830003 18400 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.009s
I20260812 16:48:07.830333 18400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "5446009ccf1544e6a244064075e68b63"
format_stamp: "Formatted at 2026-08-12 16:48:07 on dist-test-slave-1vmg"
server_key: "e56f0e69058297281729e1ffea214182"
server_key_iv: "55af6f7ffb9fa57f63d040740357476c"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:07.830664 18400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:07.904265 18400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:07.911744 18400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:07.912317 18400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:07.920477 18400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:07.929368 18400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:07.929584 18400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 16:48:07.929797 18400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:07.929909 18400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:48:07 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553287, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
Aug 12 16:48:08 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553287, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:08.584342 18400 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:37655
I20260812 16:48:08.586566 18400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:48:08.584393 18626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:37655 every 8 connection(s)
I20260812 16:48:08.595983 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 18400
I20260812 16:48:08.596262 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:48:08.597297 29644 external_mini_cluster.cc:1428] Setting key cf4524432fa8bd023d03cbd5c00b6ba8
Aug 12 16:48:08 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553287, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:08.645236 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:08.593510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36643 (local address 127.28.243.1:60363)
0812 16:48:08.607862 (+ 14352us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:08.607957 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:08.610984 (+ 3027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:08.612569 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:08.612583 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:08.613293 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:08.614208 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:08.614433 (+ 225us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:08.616788 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:08.616794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:08.619094 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:08.619128 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:08.620740 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:08.620748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:08.621081 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:08.631274 (+ 10193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:08.631336 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:08.634748 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:08.639683 (+ 4935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:08.639695 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:08.639730 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:08.640488 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:08.641069 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:08.641074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:08.641078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:08.641372 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:08.642117 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:08.642127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:08.642650 (+ 523us) client_negotiation.cc:769] Sending connection context
0812 16:48:08.643024 (+ 374us) client_negotiation.cc:240] Negotiation successful
0812 16:48:08.643629 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13866,"thread_start_us":13551,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
I20260812 16:48:08.647817 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:08.594362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:60363 (local address 127.28.243.62:36643)
0812 16:48:08.611907 (+ 17545us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:08.611913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:08.611930 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:48:08.611986 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:08.611990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:08.612063 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:08.612172 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:08.614904 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:08.616400 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:08.619403 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:08.620120 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:08.620268 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:08.635171 (+ 14903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:08.635180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:08.635185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:08.635240 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:08.639424 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:08.640703 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:08.640709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:08.640712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:08.640919 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:08.641536 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:08.641541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:08.641544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:08.641953 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:08.646656 (+ 4703us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:08.646906 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 16:48:08.647343 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17413,"thread_start_us":17254,"threads_started":1}
I20260812 16:48:08.651011 18627 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
I20260812 16:48:08.651521 18627 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:08.652673 18627 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:08.656111 18229 ts_manager.cc:194] Registered new tserver with Master: 5446009ccf1544e6a244064075e68b63 (127.28.243.1:37655)
I20260812 16:48:08.658826 18229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:60363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
W20260812 16:48:08.702739 29644 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.243.2: real 0.102s user 0.001s sys 0.028s
I20260812 16:48:08.703826 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36643
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:09.095903 18636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:09.096276 18636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:09.096439 18636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:09.104194 18636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:09.104347 18636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:09.104503 18636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:48:09.116807 18636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.243.20:33957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:09.118469 18636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:09.119998 18636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:09.135531 18660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:09.135588 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:09.135558 18657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:09.136356 18636 server_base.cc:1034] running on GCE node
I20260812 16:48:09.137157 18636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:09.138362 18636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:09.139614 18636 hybrid_clock.cc:648] HybridClock initialized: now 1786553289139559 us; error 49 us; skew 500 ppm
I20260812 16:48:09.140017 18636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:09.143651 18636 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:48:09.145409 18636 webserver.cc:460] Webserver started at http://127.28.243.2:37855/ using document root <none> and password file <none>
I20260812 16:48:09.146214 18636 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:09.146355 18636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:09.146698 18636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:09.166442 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "967382b06cdf4d9c9e96031d023a3b2b"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "4d90ea152a4f33fbe819568f4420d3e1"
server_key_iv: "477489a8ff897ecfb5ed1874a1aa9f20"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:09.167837 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "967382b06cdf4d9c9e96031d023a3b2b"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "4d90ea152a4f33fbe819568f4420d3e1"
server_key_iv: "477489a8ff897ecfb5ed1874a1aa9f20"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:09.175237 18636 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 16:48:09.190186 18683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:09.191809 18636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:48:09.192021 18636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "967382b06cdf4d9c9e96031d023a3b2b"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "4d90ea152a4f33fbe819568f4420d3e1"
server_key_iv: "477489a8ff897ecfb5ed1874a1aa9f20"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:09.192265 18636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:09.210883 18636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:09.215612 18636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:09.215947 18636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:09.217051 18636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:09.218631 18636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:09.218719 18636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:09.218840 18636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:09.218897 18636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:09.256763 18636 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:43865
I20260812 16:48:09.256814 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:43865 every 8 connection(s)
I20260812 16:48:09.258186 18636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:48:09.265055 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 18636
I20260812 16:48:09.265255 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:48:09.265694 29644 external_mini_cluster.cc:1428] Setting key 67bac03f006519d1c2337ca56e0af9cb
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:09.283352 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:09.260959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:52597 (local address 127.28.243.62:36643)
0812 16:48:09.261506 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:09.261513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:09.262985 (+ 1472us) server_negotiation.cc:407] Connection header received
0812 16:48:09.265100 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:09.265105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:09.265202 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:09.265338 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:09.267129 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.268026 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.270236 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.270657 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.270776 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:09.275574 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:09.275589 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:09.275592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:09.275649 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:09.279602 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:09.280662 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:09.280668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:09.280672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:09.280830 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:09.281336 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:09.281341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:09.281345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:09.281682 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:09.281848 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:09.282642 (+ 794us) server_negotiation.cc:299] Negotiation successful
0812 16:48:09.282947 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":250,"threads_started":1}
I20260812 16:48:09.284425 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:09.261472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36643 (local address 127.28.243.2:52597)
0812 16:48:09.262720 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:09.262780 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:09.264675 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:09.265640 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:09.265650 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:09.266132 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:09.266947 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:09.266974 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.268191 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.268195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:09.270048 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:09.270077 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.270965 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.270975 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:09.271249 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:09.272165 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:09.272219 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:09.275178 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:09.279859 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:09.279870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:09.279889 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:09.280494 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:09.280955 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:09.280959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:09.280962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:09.281195 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:09.281838 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:09.281847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:09.282217 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 16:48:09.282495 (+ 278us) client_negotiation.cc:240] Negotiation successful
0812 16:48:09.283140 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":468,"threads_started":1}
I20260812 16:48:09.286208 18891 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36643
I20260812 16:48:09.286578 18891 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:09.287813 18891 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:09.290338 18229 ts_manager.cc:194] Registered new tserver with Master: 967382b06cdf4d9c9e96031d023a3b2b (127.28.243.2:43865)
I20260812 16:48:09.291554 18229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:52597
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:48:09.343876 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36643
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:09.664034 18627 heartbeater.cc:503] Master 127.28.243.62:36643 was elected leader, sending a full tablet report...
W20260812 16:48:09.732379 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:09.732707 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:09.732810 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:09.740602 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:09.740747 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:09.740872 18901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:48:09.751403 18901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.243.20:33957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:09.752800 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:09.754184 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:09.769078 18908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:09.769148 18912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:09.769119 18909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:09.769955 18901 server_base.cc:1034] running on GCE node
I20260812 16:48:09.770644 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:09.771822 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:09.773160 18901 hybrid_clock.cc:648] HybridClock initialized: now 1786553289773051 us; error 53 us; skew 500 ppm
I20260812 16:48:09.773808 18901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:09.778419 18901 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:48:09.780532 18901 webserver.cc:460] Webserver started at http://127.28.243.3:36109/ using document root <none> and password file <none>
I20260812 16:48:09.781383 18901 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:09.781531 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:09.781881 18901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:09.798656 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "6cb325a8a89a4f68b599486de4e0c5c6"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "600f5cecbbca1c3539ffcfa33222ce27"
server_key_iv: "286fd80e1f42a2170f8e804dfe0d6b5d"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:09.799837 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "6cb325a8a89a4f68b599486de4e0c5c6"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "600f5cecbbca1c3539ffcfa33222ce27"
server_key_iv: "286fd80e1f42a2170f8e804dfe0d6b5d"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:09.806023 18901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:48:09.823680 18926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:09.825822 18901 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:48:09.826085 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "6cb325a8a89a4f68b599486de4e0c5c6"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "600f5cecbbca1c3539ffcfa33222ce27"
server_key_iv: "286fd80e1f42a2170f8e804dfe0d6b5d"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:09.826342 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:09.841817 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:09.847008 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:09.847337 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:09.848465 18901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:09.850103 18901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:09.850195 18901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:09.850327 18901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:09.850383 18901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:09.887825 18901 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:43841
I20260812 16:48:09.887889 19124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:43841 every 8 connection(s)
I20260812 16:48:09.889268 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:48:09.890837 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 18901
I20260812 16:48:09.891065 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:48:09.891687 29644 external_mini_cluster.cc:1428] Setting key 4a2576c691e0361f13d5e5891808e40d
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:09.918210 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:09.892583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:55781 (local address 127.28.243.62:36643)
0812 16:48:09.894690 (+ 2107us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:09.894695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:09.894716 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:48:09.897520 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:09.897526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:09.897653 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:09.897786 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:09.900661 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.901623 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.904371 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.905138 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.905346 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:09.910501 (+ 5155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:09.910507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:09.910510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:09.910563 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:09.913552 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:09.915066 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:09.915072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:09.915074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:09.915271 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:09.916024 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:09.916029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:09.916031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:09.916402 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:09.916549 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:09.917604 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0812 16:48:09.917928 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1992,"thread_start_us":1888,"threads_started":1}
I20260812 16:48:09.919245 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:09.893143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36643 (local address 127.28.243.3:55781)
0812 16:48:09.894185 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:09.894272 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:09.897164 (+ 2892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:09.898543 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:09.898565 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:09.899376 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:09.900327 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:09.900356 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.901967 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.901972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:09.904051 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:09.904077 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.905628 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.905638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:09.905980 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:09.907154 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:09.907193 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:09.910223 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:09.913880 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:09.913895 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:09.913929 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:09.914746 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:09.915446 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:09.915451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:09.915454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:09.915810 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:09.916634 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:09.916643 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:09.917124 (+ 481us) client_negotiation.cc:769] Sending connection context
0812 16:48:09.917435 (+ 311us) client_negotiation.cc:240] Negotiation successful
0812 16:48:09.917948 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":386,"threads_started":1}
I20260812 16:48:09.921545 19125 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36643
I20260812 16:48:09.922034 19125 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:09.923161 19125 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:09.925315 18229 ts_manager.cc:194] Registered new tserver with Master: 6cb325a8a89a4f68b599486de4e0c5c6 (127.28.243.3:43841)
I20260812 16:48:09.925968 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:48:09.926384 18229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:55781
I20260812 16:48:09.928006 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 18400
Aug 12 16:48:09 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:48:09.950578 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:09.929004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.28.243.1:37655)
0812 16:48:09.929890 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:09.929899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:09.929929 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:48:09.930006 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:09.930015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:09.930310 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:09.930520 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:09.932956 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.934298 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.936163 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:09.936947 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:09.937081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:09.943750 (+ 6669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:09.943778 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:09.943804 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:09.943873 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:09.947195 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:09.948124 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:09.948132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:09.948134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:09.948317 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:09.948834 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:09.948839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:09.948841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:09.949277 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:09.949627 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:09.949958 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 16:48:09.950261 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":420,"threads_started":1}
I20260812 16:48:09.952049 18620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:10.041005 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 18400
I20260812 16:48:10.077484 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 18636
Aug 12 16:48:10 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:48:10.105690 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:10.082967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.28.243.2:43865)
0812 16:48:10.084020 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:10.084031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:10.084078 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:48:10.084172 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:10.084180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:10.084524 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:10.084785 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:10.086204 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:10.087907 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:10.090183 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:10.090892 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:10.091076 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:10.096642 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:10.096661 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:10.096679 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:10.096747 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:10.100753 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:10.101956 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:10.101966 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:10.101968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:10.102184 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:10.102812 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:10.102818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:10.102820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:10.103594 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:10.104052 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:10.104651 (+ 599us) server_negotiation.cc:299] Negotiation successful
0812 16:48:10.105192 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":576,"threads_started":1}
I20260812 16:48:10.107524 18879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:10.192714 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 18636
I20260812 16:48:10.224771 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 18901
Aug 12 16:48:10 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:48:10.248004 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:10.226237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.28.243.3:43841)
0812 16:48:10.227223 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:10.227231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:10.227271 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:48:10.227449 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:10.227460 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:10.227764 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:10.227988 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:10.229648 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:10.231030 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:10.232816 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:10.233526 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:10.233667 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:10.239741 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:10.239773 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:10.239793 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:10.239849 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:10.243684 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:10.244740 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:10.244749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:10.244751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:10.244926 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:10.245537 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:10.245545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:10.245548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:10.246183 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:10.246584 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:10.247123 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 16:48:10.247542 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":435,"threads_started":1}
I20260812 16:48:10.249682 19116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:10.336313 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 18901
I20260812 16:48:10.365041 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 18182
I20260812 16:48:10.366904 18308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:10.484666 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 18182
I20260812 16:48:10.523594 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36643
--webserver_interface=127.28.243.62
--webserver_port=33667
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:36643
--ranger_config_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:10.961509 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:10.962303 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:10.962850 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:10.972177 19146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:10.972340 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:10.972393 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:10.972474 19146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:10.972538 19146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:10.985810 19146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:36643
--ranger_config_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=33667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:10.988148 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:10.990167 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:11.006516 19155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:11.006590 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:11.006556 19157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:11.007875 19146 server_base.cc:1034] running on GCE node
I20260812 16:48:11.009194 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:11.011631 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:11.012990 19146 hybrid_clock.cc:648] HybridClock initialized: now 1786553291012919 us; error 68 us; skew 500 ppm
I20260812 16:48:11.013494 19146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:11 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553291, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:11.022996 19146 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:11.025452 19146 webserver.cc:460] Webserver started at http://127.28.243.62:33667/ using document root <none> and password file <none>
I20260812 16:48:11.026472 19146 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:11.026645 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:48:11 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553291, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:11.078424 19146 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:48:11.082670 19172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:11.084870 19146 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260812 16:48:11.085145 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "fda91b48b0e44fbfa46df437bb1b30a9"
format_stamp: "Formatted at 2026-08-12 16:48:05 on dist-test-slave-1vmg"
server_key: "8c1c82e481ea549c0f8e844d1b0456d6"
server_key_iv: "b7f097e0177dbc879047d60f89a4227f"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:11.086587 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:11.117811 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:11.123165 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:11.123478 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:11.145056 19264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:36643 every 8 connection(s)
I20260812 16:48:11.145076 19146 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:36643
I20260812 16:48:11.147670 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:11.154400 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 19146
I20260812 16:48:11.155599 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:37655
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:36643
--webserver_port=46563
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:11.156329 19265 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:48:11.161461 19265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Bootstrap starting.
I20260812 16:48:11.183247 19265 log.cc:824] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:11.193472 19265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:48:11.194036 19265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Bootstrap complete.
I20260812 16:48:11.204358 19265 raft_consensus.cc:348] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:11.205330 19265 raft_consensus.cc:729] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda91b48b0e44fbfa46df437bb1b30a9, State: Initialized, Role: FOLLOWER
I20260812 16:48:11.206099 19265 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:11.206350 19265 raft_consensus.cc:388] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:11.206466 19265 raft_consensus.cc:482] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:11.206650 19265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:48:11.211694 19265 raft_consensus.cc:504] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:11.212778 19265 leader_election.cc:302] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda91b48b0e44fbfa46df437bb1b30a9; no voters:
I20260812 16:48:11.213814 19265 leader_election.cc:288] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:48:11.213963 19273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:48:11.216701 19265 sys_catalog.cc:564] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:11.220654 19273 raft_consensus.cc:686] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [term 2 LEADER]: Becoming Leader. State: Replica: fda91b48b0e44fbfa46df437bb1b30a9, State: Running, Role: LEADER
I20260812 16:48:11.221534 19273 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } }
I20260812 16:48:11.224965 19277 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } } }
I20260812 16:48:11.225441 19277 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:11.224855 19276 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda91b48b0e44fbfa46df437bb1b30a9. Latest consensus state: current_term: 2 leader_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda91b48b0e44fbfa46df437bb1b30a9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 36643 } } }
I20260812 16:48:11.225788 19276 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:11.227531 19299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:11.244269 19299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:11.245270 19265 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 16:48:11.246292 19299 catalog_manager.cc:1259] Loaded cluster ID: 5792fe0ec5104c30889b5441d1e7e9c9
I20260812 16:48:11.246685 19299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:11.252270 19299 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:11.254453 19299 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Loaded TSK: 0
I20260812 16:48:11.256501 19299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:48:11.669251 19268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:11.669739 19268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:11.669922 19268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:11.680639 19268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:11.680802 19268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:11.680987 19268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:48:11.699385 19268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:37655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=46563
--tserver_master_addrs=127.28.243.62:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:11.701843 19268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:11.704470 19268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:11.738781 19334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:11.742738 19338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:11.742991 19331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:11.744009 19268 server_base.cc:1034] running on GCE node
I20260812 16:48:11.744750 19268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:11.746044 19268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:11.747289 19268 hybrid_clock.cc:648] HybridClock initialized: now 1786553291747219 us; error 60 us; skew 500 ppm
I20260812 16:48:11.747649 19268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:11 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553291, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:11.751962 19268 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:48:11.754887 19268 webserver.cc:460] Webserver started at http://127.28.243.1:46563/ using document root <none> and password file <none>
I20260812 16:48:11.755805 19268 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:11.755975 19268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:48:11 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553291, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:11.797307 19268 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.009s sys 0.000s
I20260812 16:48:11.803591 19354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:11.805884 19268 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260812 16:48:11.806147 19268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "5446009ccf1544e6a244064075e68b63"
format_stamp: "Formatted at 2026-08-12 16:48:07 on dist-test-slave-1vmg"
server_key: "e56f0e69058297281729e1ffea214182"
server_key_iv: "55af6f7ffb9fa57f63d040740357476c"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:11.807767 19268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:11.861063 19268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:11.867753 19268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:11.868137 19268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:11.874492 19268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:11.878329 19268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:11.878495 19268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:11.878626 19268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:11.878703 19268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:11.932462 19551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:37655 every 8 connection(s)
I20260812 16:48:11.932523 19268 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:37655
I20260812 16:48:11.934957 19268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:48:11.940109 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 19268
I20260812 16:48:11.941051 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:43865
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:36643
--webserver_port=37855
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:48:12 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553291, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:12.019968 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:11.943612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:37165 (local address 127.28.243.62:36643)
0812 16:48:11.976783 (+ 33171us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:11.976811 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:11.983066 (+ 6255us) server_negotiation.cc:407] Connection header received
0812 16:48:11.986599 (+ 3533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:11.986618 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:11.987263 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:11.987725 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:11.993239 (+ 5514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:11.995478 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:11.999128 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:12.000456 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:12.000682 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:12.006444 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:12.006460 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:12.006488 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:12.006559 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:12.011623 (+ 5064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:12.012813 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:12.012824 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:12.012833 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:12.013071 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:12.013648 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:12.013653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:12.013657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:12.014450 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:12.014640 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:12.017915 (+ 3275us) server_negotiation.cc:299] Negotiation successful
0812 16:48:12.018459 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32709,"thread_start_us":33096,"threads_started":1}
I20260812 16:48:12.021456 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:11.942077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36643 (local address 127.28.243.1:37165)
0812 16:48:11.980259 (+ 38182us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:11.980340 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:11.985827 (+ 5487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:11.989090 (+ 3263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:11.989108 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:11.990022 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:11.992871 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:11.992929 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:11.995858 (+ 2929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:11.995866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:11.998665 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:11.998700 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:12.001299 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:12.001311 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:12.001647 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:12.002572 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:12.002610 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:12.005947 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:12.011895 (+ 5948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:12.011907 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:12.011927 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:12.012610 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:12.013231 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:12.013237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:12.013241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:12.013511 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:12.014735 (+ 1224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:12.014745 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:12.015308 (+ 563us) client_negotiation.cc:769] Sending connection context
0812 16:48:12.015637 (+ 329us) client_negotiation.cc:240] Negotiation successful
0812 16:48:12.016306 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35418,"thread_start_us":41446,"threads_started":1}
I20260812 16:48:12.025913 19552 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36643
I20260812 16:48:12.026283 19552 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:12.027334 19552 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
W20260812 16:48:12.031422 19552 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:36643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 16:48:12.031709 19552 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:12.032047 19552 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:12.033936 19552 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:12.034365 19552 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
W20260812 16:48:12.035955 19552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:48:12.468299 19562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:12.468667 19562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:12.469099 19562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:12.477113 19562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:12.477283 19562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:12.477452 19562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:48:12.490873 19562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:43865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=37855
--tserver_master_addrs=127.28.243.62:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:12.492520 19562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:12.494015 19562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:12.510452 19590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:12.510182 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:12.510000 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:12.511761 19562 server_base.cc:1034] running on GCE node
I20260812 16:48:12.512840 19562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:12.514230 19562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:12.515588 19562 hybrid_clock.cc:648] HybridClock initialized: now 1786553292515513 us; error 51 us; skew 500 ppm
I20260812 16:48:12.516189 19562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:12 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:12.521317 19562 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:48:12.523409 19562 webserver.cc:460] Webserver started at http://127.28.243.2:37855/ using document root <none> and password file <none>
I20260812 16:48:12.524317 19562 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:12.524487 19562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:48:12 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:12.564182 19562 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.010s sys 0.001s
I20260812 16:48:12.567559 19614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:12.569551 19562 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 16:48:12.569855 19562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "967382b06cdf4d9c9e96031d023a3b2b"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "4d90ea152a4f33fbe819568f4420d3e1"
server_key_iv: "477489a8ff897ecfb5ed1874a1aa9f20"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:12.571593 19562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:12.624408 19562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:12.629742 19562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:12.630069 19562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:12.631389 19562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:12.633067 19562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:12.633185 19562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:12.633308 19562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:12.633378 19562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:48:12 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786553292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:12.712855 19562 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:43865
I20260812 16:48:12.712904 19827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:43865 every 8 connection(s)
I20260812 16:48:12.715309 19562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:48:12.716656 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 19562
I20260812 16:48:12.717473 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.20:33957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:43841
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:36643
--webserver_port=36109
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:48:12 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:12.749152 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:12.721654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:41865 (local address 127.28.243.62:36643)
0812 16:48:12.727472 (+ 5818us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:12.727479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:12.727508 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:48:12.727587 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:12.727593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:12.727713 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:12.727887 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:12.730267 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:12.731379 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:12.733990 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:12.735259 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:12.735400 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:12.740714 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:12.740722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:12.740728 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:12.740790 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:12.744625 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:12.745852 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:12.745859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:12.745863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:12.746055 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:12.746691 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:12.746697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:12.746701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:12.747163 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:12.747330 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:12.748444 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0812 16:48:12.748749 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5634,"thread_start_us":5468,"threads_started":1}
I20260812 16:48:12.750648 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:12.720451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36643 (local address 127.28.243.2:41865)
0812 16:48:12.721958 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:12.722041 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:12.724666 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:12.728304 (+ 3638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:12.728318 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:12.729016 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:12.729971 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:12.730002 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:12.731625 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:12.731631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:12.733741 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:12.733764 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:12.735772 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:12.735782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:12.736159 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:12.737197 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:12.737241 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:12.740343 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:12.744893 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:12.744906 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:12.744931 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:12.745673 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:12.746207 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:12.746212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:12.746215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:12.746543 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:12.747348 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:12.747358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:12.747890 (+ 532us) client_negotiation.cc:769] Sending connection context
0812 16:48:12.748310 (+ 420us) client_negotiation.cc:240] Negotiation successful
0812 16:48:12.748979 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"thread_start_us":705,"threads_started":1}
I20260812 16:48:12.753135 19828 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36643
I20260812 16:48:12.753533 19828 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:12.754606 19828 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:12.757190 19204 ts_manager.cc:194] Registered new tserver with Master: 967382b06cdf4d9c9e96031d023a3b2b (127.28.243.2:43865)
I20260812 16:48:12.759624 19204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:41865
I20260812 16:48:13.036155 19552 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:13.036778 19552 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:13.038851 19204 ts_manager.cc:194] Registered new tserver with Master: 5446009ccf1544e6a244064075e68b63 (127.28.243.1:37655)
I20260812 16:48:13.040441 19204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:37165
Aug 12 16:48:13 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
W20260812 16:48:13.237955 19839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:13.238549 19839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:13.238776 19839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:13.250885 19839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:13.251192 19839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:13.251473 19839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:48:13.268401 19839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.20:33957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:43841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=36109
--tserver_master_addrs=127.28.243.62:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:13.270623 19839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:13.272691 19839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:13.325407 19848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:13.326942 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:13.360932 19858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:13.362057 19839 server_base.cc:1034] running on GCE node
I20260812 16:48:13.363042 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:13.373229 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:13.377650 19839 hybrid_clock.cc:648] HybridClock initialized: now 1786553293377495 us; error 132 us; skew 500 ppm
I20260812 16:48:13.378419 19839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:13 dist-test-slave-1vmg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:13.383903 19839 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:48:13.411194 19839 webserver.cc:460] Webserver started at http://127.28.243.3:36109/ using document root <none> and password file <none>
I20260812 16:48:13.412129 19839 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:13.412307 19839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:48:13 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:13.436901 19839 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:48:13.440193 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:13.441859 19839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:48:13.442059 19839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "6cb325a8a89a4f68b599486de4e0c5c6"
format_stamp: "Formatted at 2026-08-12 16:48:09 on dist-test-slave-1vmg"
server_key: "600f5cecbbca1c3539ffcfa33222ce27"
server_key_iv: "286fd80e1f42a2170f8e804dfe0d6b5d"
server_key_version: "kuduclusterkey@0"
I20260812 16:48:13.443246 19839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:13.470239 19839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:13.475101 19839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:13.475416 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:13.476630 19839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:13.478572 19839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:13.478693 19839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:13.478824 19839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:13.478897 19839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:48:13 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for HTTP/127.28.243.20@KRBTEST.COM
I20260812 16:48:13.638968 19839 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:43841
I20260812 16:48:13.639106 20058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:43841 every 8 connection(s)
I20260812 16:48:13.641328 19839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:48:13.647244 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 19839
Aug 12 16:48:13 dist-test-slave-1vmg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:13.704200 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.647282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:52001 (local address 127.28.243.62:36643)
0812 16:48:13.674944 (+ 27662us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:13.674953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:13.674989 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:48:13.675088 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:13.675095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:13.675202 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:13.675391 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:13.679096 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.680344 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.686983 (+ 6639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.687770 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.687938 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.695029 (+ 7091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:13.695037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:13.695043 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:13.695104 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:13.699502 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:13.700838 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:13.700844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:13.700848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:13.701045 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:13.701734 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:13.701738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:13.701741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:13.702157 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:13.702299 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:13.703469 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0812 16:48:13.703753 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":27433,"thread_start_us":27273,"threads_started":1}
I20260812 16:48:13.707432 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.646844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:36643 (local address 127.28.243.3:52001)
0812 16:48:13.660223 (+ 13379us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:13.660348 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:13.663409 (+ 3061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:13.676200 (+ 12791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:13.676219 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:13.677177 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:13.678609 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.678651 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.680665 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.680671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.683046 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.683085 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.688529 (+ 5444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.688539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.688941 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.691022 (+ 2081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:13.691063 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:13.694596 (+ 3533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:13.699781 (+ 5185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:13.699797 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:13.699822 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:13.700647 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:13.701232 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:13.701237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:13.701241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:13.701576 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:13.702330 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:13.702340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:13.702873 (+ 533us) client_negotiation.cc:769] Sending connection context
0812 16:48:13.704950 (+ 2077us) client_negotiation.cc:240] Negotiation successful
0812 16:48:13.705692 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12803,"thread_start_us":11864,"threads_started":1}
I20260812 16:48:13.709681 20059 heartbeater.cc:347] Connected to a master server at 127.28.243.62:36643
I20260812 16:48:13.710109 20059 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:13.711201 20059 heartbeater.cc:511] Master 127.28.243.62:36643 requested a full tablet report, sending...
I20260812 16:48:13.711586 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.674856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.28.243.62:36643)
0812 16:48:13.690162 (+ 15306us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:13.690168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:13.690229 (+ 61us) server_negotiation.cc:407] Connection header received
0812 16:48:13.690303 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:13.690308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:13.690403 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:13.690559 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:13.693051 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.694223 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.697259 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.698044 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.702359 (+ 4315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.702659 (+ 300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:13.702666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:13.702670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:13.702703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:13.706325 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:13.707457 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:13.707462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:13.707465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:13.707662 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:13.708556 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:13.708564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:13.708569 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:13.709156 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:13.709600 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:13.710832 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0812 16:48:13.711192 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":15104,"thread_start_us":14737,"threads_started":1}
I20260812 16:48:13.717878 19204 ts_manager.cc:194] Registered new tserver with Master: 6cb325a8a89a4f68b599486de4e0c5c6 (127.28.243.3:43841)
I20260812 16:48:13.719250 19204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:52001
I20260812 16:48:13.724242 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:48:13.767494 19828 heartbeater.cc:503] Master 127.28.243.62:36643 was elected leader, sending a full tablet report...
I20260812 16:48:13.784005 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.752214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.28.243.62:36643)
0812 16:48:13.752455 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:13.752462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:13.761110 (+ 8648us) server_negotiation.cc:407] Connection header received
0812 16:48:13.763433 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:13.763441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:13.763563 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:13.763658 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:13.766254 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.768426 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.770814 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.771533 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.771655 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.775828 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:13.775836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:13.775842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:13.775896 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:13.780066 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:13.781234 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:13.781241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:13.781244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:13.781453 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:13.781967 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:13.781973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:13.781977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:13.782381 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:13.782636 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:13.783349 (+ 713us) server_negotiation.cc:299] Negotiation successful
0812 16:48:13.783642 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:48:13.790872 19204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:48:13.803707 19204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:48:13.868444 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.843924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.28.243.2:43865)
0812 16:48:13.857316 (+ 13392us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:13.857326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:13.857512 (+ 186us) server_negotiation.cc:407] Connection header received
0812 16:48:13.857662 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:13.857672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:13.858085 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:13.858341 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:13.859442 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.861138 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.864469 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.866245 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.866390 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.866695 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:13.867771 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0812 16:48:13.868013 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13204,"thread_start_us":13012,"threads_started":1}
I20260812 16:48:13.869823 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.843349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:43865 (local address 127.0.0.1:41782)
0812 16:48:13.857286 (+ 13937us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:13.857308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:13.857472 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:13.858663 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:13.858670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:13.858703 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:13.859184 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.859200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.861386 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.861392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.864205 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.864230 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.867028 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.867035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.867382 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.867449 (+ 67us) client_negotiation.cc:769] Sending connection context
0812 16:48:13.869101 (+ 1652us) client_negotiation.cc:240] Negotiation successful
0812 16:48:13.869386 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13806,"thread_start_us":12107,"threads_started":1}
I20260812 16:48:13.875262 19711 tablet_service.cc:1467] Processing CreateTablet for tablet 5aeff2e3eded45449f0c3d8789659bb1 (DEFAULT_TABLE table=test-table [id=2d208d4a011941fe8e83cf5173b66a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:13.877416 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aeff2e3eded45449f0c3d8789659bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:13.879293 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.842789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:43841 (local address 127.0.0.1:44628)
0812 16:48:13.856156 (+ 13367us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:13.856225 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:13.856539 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:13.859532 (+ 2993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:13.859537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:13.859575 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:13.860319 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.860335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.865887 (+ 5552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.865895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.868423 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.868445 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.877550 (+ 9105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.877561 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.877840 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.877892 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 16:48:13.877996 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 16:48:13.878188 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12930,"thread_start_us":14760,"threads_started":1}
I20260812 16:48:13.881650 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.843287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.28.243.3:43841)
0812 16:48:13.857881 (+ 14594us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:13.857889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:13.857924 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:48:13.858219 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:13.858230 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:13.858697 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:13.859172 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:13.860586 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.862405 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.868743 (+ 6338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.870586 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.880520 (+ 9934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.880871 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:13.880976 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 16:48:13.881224 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14403,"thread_start_us":14226,"threads_started":1}
I20260812 16:48:13.889097 19949 tablet_service.cc:1467] Processing CreateTablet for tablet 5aeff2e3eded45449f0c3d8789659bb1 (DEFAULT_TABLE table=test-table [id=2d208d4a011941fe8e83cf5173b66a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:13.891502 19949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aeff2e3eded45449f0c3d8789659bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:13.895332 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.860177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:37655 (local address 127.0.0.1:58018)
0812 16:48:13.879938 (+ 19761us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:13.880054 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:13.880248 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:13.883435 (+ 3187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:13.883441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:13.883479 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:13.883705 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.883717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.887105 (+ 3388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.887111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.889710 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:13.889739 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.894065 (+ 4326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.894074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:13.894365 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.894487 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 16:48:13.894609 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 16:48:13.894971 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19591,"thread_start_us":23207,"threads_started":1}
I20260812 16:48:13.898797 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:13.858695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.28.243.1:37655)
0812 16:48:13.882543 (+ 23848us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:13.882555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:13.882597 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:48:13.882734 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:13.882744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:13.883117 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:13.883903 (+ 786us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:13.884195 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.885780 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.890296 (+ 4516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:13.892368 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:13.895855 (+ 3487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:13.896186 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:13.896288 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 16:48:13.896480 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23613,"thread_start_us":23380,"threads_started":1}
I20260812 16:48:13.903579 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 5aeff2e3eded45449f0c3d8789659bb1 (DEFAULT_TABLE table=test-table [id=2d208d4a011941fe8e83cf5173b66a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:13.905925 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aeff2e3eded45449f0c3d8789659bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:13.906677 20105 tablet_bootstrap.cc:492] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Bootstrap starting.
I20260812 16:48:13.913288 20109 tablet_bootstrap.cc:492] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Bootstrap starting.
I20260812 16:48:13.916942 20105 tablet_bootstrap.cc:654] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:13.918598 20105 log.cc:824] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:13.919162 20109 tablet_bootstrap.cc:654] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:13.920686 20109 log.cc:824] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:13.926527 20105 tablet_bootstrap.cc:492] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: No bootstrap required, opened a new log
I20260812 16:48:13.927049 20105 ts_tablet_manager.cc:1397] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.005s
I20260812 16:48:13.931169 20113 tablet_bootstrap.cc:492] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Bootstrap starting.
I20260812 16:48:13.938087 20113 tablet_bootstrap.cc:654] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:13.938292 20109 tablet_bootstrap.cc:492] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: No bootstrap required, opened a new log
I20260812 16:48:13.938614 20109 ts_tablet_manager.cc:1397] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.011s
I20260812 16:48:13.939627 20113 log.cc:824] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:13.942699 20105 raft_consensus.cc:348] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:13.943461 20105 raft_consensus.cc:374] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:13.943636 20105 raft_consensus.cc:729] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967382b06cdf4d9c9e96031d023a3b2b, State: Initialized, Role: FOLLOWER
I20260812 16:48:13.944612 20105 consensus_queue.cc:260] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:13.951479 20109 raft_consensus.cc:348] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:13.952966 20109 raft_consensus.cc:374] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:13.953123 20109 raft_consensus.cc:729] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb325a8a89a4f68b599486de4e0c5c6, State: Initialized, Role: FOLLOWER
I20260812 16:48:13.953958 20109 consensus_queue.cc:260] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:13.956198 20105 ts_tablet_manager.cc:1428] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Time spent starting tablet: real 0.029s user 0.014s sys 0.009s
I20260812 16:48:13.958415 20113 tablet_bootstrap.cc:492] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: No bootstrap required, opened a new log
I20260812 16:48:13.959812 20113 ts_tablet_manager.cc:1397] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.011s
I20260812 16:48:13.963733 20059 heartbeater.cc:503] Master 127.28.243.62:36643 was elected leader, sending a full tablet report...
I20260812 16:48:13.964540 20109 ts_tablet_manager.cc:1428] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Time spent starting tablet: real 0.026s user 0.016s sys 0.008s
I20260812 16:48:13.970329 20113 raft_consensus.cc:348] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:13.970846 20113 raft_consensus.cc:374] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:13.970989 20113 raft_consensus.cc:729] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5446009ccf1544e6a244064075e68b63, State: Initialized, Role: FOLLOWER
W20260812 16:48:13.971585 19831 tablet.cc:2369] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:13.971701 20113 consensus_queue.cc:260] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:13.972874 19552 heartbeater.cc:503] Master 127.28.243.62:36643 was elected leader, sending a full tablet report...
I20260812 16:48:13.975713 20113 ts_tablet_manager.cc:1428] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260812 16:48:13.987715 19553 tablet.cc:2369] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:48:14.144640 20061 tablet.cc:2369] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:14.211751 20115 raft_consensus.cc:482] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:48:14.212179 20115 raft_consensus.cc:504] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:14.214787 20115 leader_election.cc:288] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cb325a8a89a4f68b599486de4e0c5c6 (127.28.243.3:43841), 5446009ccf1544e6a244064075e68b63 (127.28.243.1:37655)
I20260812 16:48:14.224723 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.215591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:56675 (local address 127.28.243.3:43841)
0812 16:48:14.215814 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:14.215821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:14.216909 (+ 1088us) server_negotiation.cc:407] Connection header received
0812 16:48:14.217535 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:14.217542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:14.217645 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:14.217763 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:14.218649 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.219719 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.222302 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.223593 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.223787 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.223869 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:14.224302 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 16:48:14.224450 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:48:14.224884 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.215541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:43841 (local address 127.28.243.2:56675)
0812 16:48:14.216765 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:14.216787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:14.216989 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:14.217996 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:14.218001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:14.218031 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:14.218454 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:14.218467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.219856 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.219861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:14.222047 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:14.222066 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.223833 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.223840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:14.224023 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.224061 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:48:14.224247 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 16:48:14.224457 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":753,"threads_started":1}
I20260812 16:48:14.225766 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.217275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:37655 (local address 127.28.243.2:52491)
0812 16:48:14.218242 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:14.218264 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:14.218469 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:14.218836 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:14.218841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:14.218871 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:14.219376 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:14.219392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.221213 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.221220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:14.223201 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:14.223216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.224955 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.224962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:14.225155 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.225187 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:48:14.225278 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 16:48:14.225417 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":484,"threads_started":1}
I20260812 16:48:14.225819 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.217354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:52491 (local address 127.28.243.1:37655)
0812 16:48:14.217555 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:14.217560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:14.218411 (+ 851us) server_negotiation.cc:407] Connection header received
0812 16:48:14.218608 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:14.218613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:14.218692 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:14.218802 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:14.219582 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.220955 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.223388 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.224522 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.224712 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.224788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:14.225333 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 16:48:14.225538 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:48:14.226372 19977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aeff2e3eded45449f0c3d8789659bb1" candidate_uuid: "967382b06cdf4d9c9e96031d023a3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" is_pre_election: true
I20260812 16:48:14.227087 19977 raft_consensus.cc:2454] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967382b06cdf4d9c9e96031d023a3b2b in term 0.
I20260812 16:48:14.227169 19474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aeff2e3eded45449f0c3d8789659bb1" candidate_uuid: "967382b06cdf4d9c9e96031d023a3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5446009ccf1544e6a244064075e68b63" is_pre_election: true
I20260812 16:48:14.227661 19474 raft_consensus.cc:2454] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967382b06cdf4d9c9e96031d023a3b2b in term 0.
I20260812 16:48:14.227909 19627 leader_election.cc:302] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb325a8a89a4f68b599486de4e0c5c6, 967382b06cdf4d9c9e96031d023a3b2b; no voters:
I20260812 16:48:14.228433 20115 raft_consensus.cc:2781] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:48:14.228670 20115 raft_consensus.cc:482] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:48:14.228843 20115 raft_consensus.cc:3037] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:14.232244 20115 raft_consensus.cc:504] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:14.232937 20115 leader_election.cc:288] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [CANDIDATE]: Term 1 election: Requested vote from peers 6cb325a8a89a4f68b599486de4e0c5c6 (127.28.243.3:43841), 5446009ccf1544e6a244064075e68b63 (127.28.243.1:37655)
I20260812 16:48:14.233700 19474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aeff2e3eded45449f0c3d8789659bb1" candidate_uuid: "967382b06cdf4d9c9e96031d023a3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5446009ccf1544e6a244064075e68b63"
I20260812 16:48:14.233675 19977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aeff2e3eded45449f0c3d8789659bb1" candidate_uuid: "967382b06cdf4d9c9e96031d023a3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb325a8a89a4f68b599486de4e0c5c6"
I20260812 16:48:14.233923 19474 raft_consensus.cc:3037] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:14.233923 19977 raft_consensus.cc:3037] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:14.236733 19474 raft_consensus.cc:2454] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967382b06cdf4d9c9e96031d023a3b2b in term 1.
I20260812 16:48:14.237253 19977 raft_consensus.cc:2454] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967382b06cdf4d9c9e96031d023a3b2b in term 1.
I20260812 16:48:14.237417 19627 leader_election.cc:302] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5446009ccf1544e6a244064075e68b63, 967382b06cdf4d9c9e96031d023a3b2b; no voters:
I20260812 16:48:14.237846 20115 raft_consensus.cc:2781] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:14.238412 20115 raft_consensus.cc:686] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 1 LEADER]: Becoming Leader. State: Replica: 967382b06cdf4d9c9e96031d023a3b2b, State: Running, Role: LEADER
I20260812 16:48:14.239542 20115 consensus_queue.cc:237] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } }
I20260812 16:48:14.246531 19204 catalog_manager.cc:5495] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 967382b06cdf4d9c9e96031d023a3b2b (127.28.243.2). New cstate: current_term: 1 leader_uuid: "967382b06cdf4d9c9e96031d023a3b2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967382b06cdf4d9c9e96031d023a3b2b" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 } health_report { overall_health: UNKNOWN } } }
I20260812 16:48:14.280085 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.272750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.28.243.2:43865)
0812 16:48:14.273103 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:14.273109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:14.273135 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:48:14.273291 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:14.273424 (+ 133us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:14.273599 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:14.273778 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:48:14.274222 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.275046 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.277181 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.277708 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.278083 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.278780 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:48:14.279420 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:48:14.279549 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:14.279764 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 16:48:14.279860 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260812 16:48:14.289587 19977 raft_consensus.cc:1261] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 1 FOLLOWER]: Refusing update from remote peer 967382b06cdf4d9c9e96031d023a3b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:14.289599 19474 raft_consensus.cc:1261] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 1 FOLLOWER]: Refusing update from remote peer 967382b06cdf4d9c9e96031d023a3b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:14.290856 20124 consensus_queue.cc:1035] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5446009ccf1544e6a244064075e68b63" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:14.291587 20115 consensus_queue.cc:1035] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb325a8a89a4f68b599486de4e0c5c6" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 43841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:14.324303 19202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:14.324767 19202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:14.329195 19202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fda91b48b0e44fbfa46df437bb1b30a9: Sending DeleteTablet for 3 replicas of tablet 5aeff2e3eded45449f0c3d8789659bb1
I20260812 16:48:14.330693 19949 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aeff2e3eded45449f0c3d8789659bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:14 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:44628
I20260812 16:48:14.330757 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aeff2e3eded45449f0c3d8789659bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:14 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:58018
I20260812 16:48:14.330752 19711 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aeff2e3eded45449f0c3d8789659bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:14 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:41782
I20260812 16:48:14.334512 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 19268
I20260812 16:48:14.343995 20155 tablet_replica.cc:330] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: stopping tablet replica
I20260812 16:48:14.344635 20155 raft_consensus.cc:2229] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:14.345326 20155 raft_consensus.cc:2258] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:14.348536 20155 ts_tablet_manager.cc:1905] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:14.357429 20155 ts_tablet_manager.cc:1918] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:14.357837 20155 log.cc:1196] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/5aeff2e3eded45449f0c3d8789659bb1
I20260812 16:48:14.359086 20155 ts_tablet_manager.cc:1939] T 5aeff2e3eded45449f0c3d8789659bb1 P 6cb325a8a89a4f68b599486de4e0c5c6: Deleting consensus metadata
I20260812 16:48:14.360015 20154 tablet_replica.cc:330] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: stopping tablet replica
I20260812 16:48:14.361032 20153 tablet_replica.cc:330] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: stopping tablet replica
I20260812 16:48:14.361449 20154 raft_consensus.cc:2229] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:48:14.361544 20153 raft_consensus.cc:2229] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:14.361673 19178 catalog_manager.cc:4841] TS 6cb325a8a89a4f68b599486de4e0c5c6 (127.28.243.3:43841): tablet 5aeff2e3eded45449f0c3d8789659bb1 (table test-table [id=2d208d4a011941fe8e83cf5173b66a85]) successfully deleted
I20260812 16:48:14.362339 20154 raft_consensus.cc:2258] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:14.362483 20153 raft_consensus.cc:2258] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:14.365043 20154 ts_tablet_manager.cc:1905] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:14.365037 20153 ts_tablet_manager.cc:1905] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:14.374646 20154 ts_tablet_manager.cc:1918] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:14.375164 20154 log.cc:1196] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/5aeff2e3eded45449f0c3d8789659bb1
I20260812 16:48:14.376035 20154 ts_tablet_manager.cc:1939] T 5aeff2e3eded45449f0c3d8789659bb1 P 967382b06cdf4d9c9e96031d023a3b2b: Deleting consensus metadata
I20260812 16:48:14.377856 20153 ts_tablet_manager.cc:1918] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:14.378404 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.335896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.28.243.1:37655)
0812 16:48:14.336097 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:14.336103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:14.339657 (+ 3554us) server_negotiation.cc:407] Connection header received
0812 16:48:14.339875 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:14.339881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:14.339978 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:14.340090 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:14.348585 (+ 8495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.349938 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.356730 (+ 6792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.357685 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.357852 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.366217 (+ 8365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:14.366233 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:14.366252 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:14.366319 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:14.372656 (+ 6337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:14.373838 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:14.373850 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:14.373855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:14.374061 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:14.374575 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:14.374580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:14.374583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:14.375293 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:14.375723 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:14.377325 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0812 16:48:14.377826 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:48:14.378414 20153 log.cc:1196] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/5aeff2e3eded45449f0c3d8789659bb1
I20260812 16:48:14.379606 19545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:14.379990 19175 catalog_manager.cc:4841] TS 967382b06cdf4d9c9e96031d023a3b2b (127.28.243.2:43865): tablet 5aeff2e3eded45449f0c3d8789659bb1 (table test-table [id=2d208d4a011941fe8e83cf5173b66a85]) successfully deleted
I20260812 16:48:14.506522 20153 ts_tablet_manager.cc:1939] T 5aeff2e3eded45449f0c3d8789659bb1 P 5446009ccf1544e6a244064075e68b63: Deleting consensus metadata
I20260812 16:48:14.509285 19178 catalog_manager.cc:4841] TS 5446009ccf1544e6a244064075e68b63 (127.28.243.1:37655): tablet 5aeff2e3eded45449f0c3d8789659bb1 (table test-table [id=2d208d4a011941fe8e83cf5173b66a85]) successfully deleted
I20260812 16:48:14.509357 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 19268
I20260812 16:48:14.543095 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 19562
I20260812 16:48:14.564198 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.544460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.28.243.2:43865)
0812 16:48:14.544701 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:14.544708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:14.544741 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:48:14.544813 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:14.544819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:14.544903 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:14.545031 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:14.547698 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.549026 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.550585 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.551531 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.551689 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.554731 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:14.554764 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:14.554792 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:14.554867 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:14.559670 (+ 4803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:14.561038 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:14.561051 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:14.561055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:14.561308 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:14.562015 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:14.562021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:14.562025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:14.562544 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:14.562910 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:14.563474 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 16:48:14.563799 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:48:14.565392 19823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:14.697077 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 19562
I20260812 16:48:14.729539 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 19839
I20260812 16:48:14.746479 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:14.730408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.28.243.3:43841)
0812 16:48:14.730956 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:14.730962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:14.730984 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:48:14.731045 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:14.731049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:14.731128 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:14.731229 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:14.732754 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.733578 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.735084 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:14.735522 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:14.735639 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:14.738551 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:14.738571 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:14.738591 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:14.738642 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:14.742370 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:14.743882 (+ 1512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:14.743892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:14.743894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:14.744080 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:14.744618 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:14.744621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:14.744623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:14.745066 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:14.745340 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:14.745838 (+ 498us) server_negotiation.cc:299] Negotiation successful
0812 16:48:14.746172 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
I20260812 16:48:14.747385 20048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:14.847826 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 19839
I20260812 16:48:14.880316 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 19146
I20260812 16:48:14.881958 19258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:15.004825 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 19146
I20260812 16:48:15.044839 29644 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 16:48:15.396838 29644 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 16:48:15.397200 29644 mini_ranger.cc:66] Stopping Ranger...
I20260812 16:48:15.756445 29644 mini_ranger.cc:68] Stopped Ranger
2026-08-12 16:48:15.756 UTC [17437] LOG: received smart shutdown request
2026-08-12 16:48:15.761 UTC [17437] LOG: background worker "logical replication launcher" (PID 17453) exited with exit code 1
2026-08-12 16:48:15.762 UTC [17442] LOG: shutting down
2026-08-12 16:48:15.980 UTC [17437] LOG: database system is shut down
2026-08-12T16:48:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (64645 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:48:16 dist-test-slave-1vmg krb5kdc[20187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:48:16 dist-test-slave-1vmg krb5kdc[20187](info): set up 2 sockets
Aug 12 16:48:16 dist-test-slave-1vmg krb5kdc[20187](info): commencing operation
krb5kdc: starting...
W20260812 16:48:17.492127 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:48:17 dist-test-slave-1vmg krb5kdc[20187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:48:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:48:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:48:17.836786 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43201
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43201,127.28.243.61:37675,127.28.243.60:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:18.214474 20212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:18.214828 20212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:18.214972 20212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:18.223424 20212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:18.223578 20212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:18.223634 20212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:18.223694 20212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:18.223773 20212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:18.234321 20212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43201,127.28.243.61:37675,127.28.243.60:42093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:18.235867 20212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:18.237282 20212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:18.251883 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:18.251945 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:18.251945 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:18.253019 20212 server_base.cc:1034] running on GCE node
I20260812 16:48:18.253958 20212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:18.255910 20212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:18.257210 20212 hybrid_clock.cc:648] HybridClock initialized: now 1786553298257141 us; error 54 us; skew 500 ppm
I20260812 16:48:18.258064 20212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:18 dist-test-slave-1vmg krb5kdc[20187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:18.262645 20212 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:18.264691 20212 webserver.cc:460] Webserver started at http://127.28.243.62:34603/ using document root <none> and password file <none>
I20260812 16:48:18.265502 20212 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:18.265645 20212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:18.265974 20212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:18.269448 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "d9295f519def4323b51339765a3bc86f"
format_stamp: "Formatted at 2026-08-12 16:48:18 on dist-test-slave-1vmg"
server_key: "37d18672ca95fbfd317a74a8015ec302"
server_key_iv: "174bdce44b5801a453fbc9be989fdb2e"
server_key_version: "encryptionkey@0"
I20260812 16:48:18.270395 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "d9295f519def4323b51339765a3bc86f"
format_stamp: "Formatted at 2026-08-12 16:48:18 on dist-test-slave-1vmg"
server_key: "37d18672ca95fbfd317a74a8015ec302"
server_key_iv: "174bdce44b5801a453fbc9be989fdb2e"
server_key_version: "encryptionkey@0"
I20260812 16:48:18.277169 20212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:48:18.283113 20252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:18.285439 20212 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 16:48:18.285699 20212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "d9295f519def4323b51339765a3bc86f"
format_stamp: "Formatted at 2026-08-12 16:48:18 on dist-test-slave-1vmg"
server_key: "37d18672ca95fbfd317a74a8015ec302"
server_key_iv: "174bdce44b5801a453fbc9be989fdb2e"
server_key_version: "encryptionkey@0"
I20260812 16:48:18.285954 20212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:18.302129 20212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:18.306862 20212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:18.307232 20212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:18.331424 20212 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:43201
I20260812 16:48:18.331468 20340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:43201 every 8 connection(s)
I20260812 16:48:18.332933 20212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:18.337392 20342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:18.342525 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 20212
I20260812 16:48:18.342779 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:48:18.343503 29644 external_mini_cluster.cc:1428] Setting key 1dfbac58e0bfd1d71b505e822b74e928
I20260812 16:48:18.346446 20342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:48:18 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.61@KRBTEST.COM" created.
I20260812 16:48:18.376186 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.351822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.28.243.62:43201)
0812 16:48:18.353893 (+ 2071us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:18.353902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:18.353965 (+ 63us) server_negotiation.cc:407] Connection header received
0812 16:48:18.355737 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:18.355744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:18.356016 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:18.356216 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:18.357628 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.359273 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.361174 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.361869 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.362093 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:18.367529 (+ 5436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:18.367550 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:18.367572 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:18.367625 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:18.371339 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:18.372237 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:18.372244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:18.372246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:18.372412 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:18.372811 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:18.372815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:18.372817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:18.373392 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:18.373604 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:18.374176 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 16:48:18.374603 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1831,"spinlock_wait_cycles":21760,"thread_start_us":1581,"threads_started":1}
I20260812 16:48:18.376274 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.351821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43201 (local address 127.0.0.1:51084)
0812 16:48:18.352895 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.352955 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:18.355300 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:18.356263 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:18.356271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:18.357043 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:18.357407 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:18.357433 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.359535 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.359541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:18.360950 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:18.360971 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.362174 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.362182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:18.362369 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:18.363381 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:18.363413 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:18.366836 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:18.371530 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:18.371545 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:18.371560 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:18.372091 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:18.372530 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:18.372535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:18.372539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:18.372708 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:18.373536 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:18.373544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:18.373787 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 16:48:18.374022 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 16:48:18.374754 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":195,"thread_start_us":468,"threads_started":1}
I20260812 16:48:18.379838 20342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:48:18.381295 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.380785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35694)
0812 16:48:18.380988 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.381132 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 16:48:18.381621 20254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
I20260812 16:48:18.384791 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.384177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35706)
0812 16:48:18.384345 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.384451 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 16:48:18.385154 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37675: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab.
Entry for principal kudu/127.28.243.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab.
I20260812 16:48:18.416306 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:37675
--webserver_interface=127.28.243.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab
--principal=kudu/127.28.243.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43201,127.28.243.61:37675,127.28.243.60:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:18.434662 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } attempt: 1
I20260812 16:48:18.436060 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.435447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35710)
0812 16:48:18.435708 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.435883 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 16:48:18.437574 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.437127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35720)
0812 16:48:18.437304 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.437400 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 16:48:18.437839 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37675: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
I20260812 16:48:18.506171 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } attempt: 2
I20260812 16:48:18.507783 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.507186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35724)
0812 16:48:18.507476 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.507595 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 16:48:18.509310 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.508742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35738)
0812 16:48:18.508939 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.509086 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 16:48:18.509630 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37675: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
I20260812 16:48:18.600972 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } attempt: 3
I20260812 16:48:18.602712 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.601984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35742)
0812 16:48:18.602317 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.602501 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260812 16:48:18.604460 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.603913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35746)
0812 16:48:18.604132 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.604250 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 16:48:18.604786 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37675: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
I20260812 16:48:18.748113 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } attempt: 4
I20260812 16:48:18.749620 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.748921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35758)
0812 16:48:18.749309 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.749429 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260812 16:48:18.751631 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.750852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35764)
0812 16:48:18.751116 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.751308 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260812 16:48:18.751914 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37675: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
W20260812 16:48:18.818274 20354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:18.818620 20354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:18.818714 20354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:18.826730 20354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:18.826880 20354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:18.826969 20354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:18.827031 20354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:18.827093 20354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:18.840242 20354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43201,127.28.243.61:37675,127.28.243.60:42093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab
--principal=kudu/127.28.243.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:18.841782 20354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:18.843276 20354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:18.857007 20361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:18.857049 20364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:18.857075 20366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:18.858618 20354 server_base.cc:1034] running on GCE node
I20260812 16:48:18.859907 20354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:18.861733 20354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:18.863013 20354 hybrid_clock.cc:648] HybridClock initialized: now 1786553298862953 us; error 75 us; skew 500 ppm
I20260812 16:48:18.863413 20354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:18 dist-test-slave-1vmg krb5kdc[20187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:18.867339 20354 init.cc:377] Logged in from keytab as kudu/127.28.243.61@KRBTEST.COM (short username kudu)
I20260812 16:48:18.869094 20354 webserver.cc:460] Webserver started at http://127.28.243.61:45189/ using document root <none> and password file <none>
I20260812 16:48:18.869887 20354 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:18.870003 20354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:18.870307 20354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:18.874285 20354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data/instance:
uuid: "2d55c01b18e840ac98253279fc5e8b34"
format_stamp: "Formatted at 2026-08-12 16:48:18 on dist-test-slave-1vmg"
server_key: "4da0e7a24d1c22603d187f84c23d3461"
server_key_iv: "c4623213ef464e638685ee0c5c7df071"
server_key_version: "encryptionkey@0"
I20260812 16:48:18.875514 20354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal/instance:
uuid: "2d55c01b18e840ac98253279fc5e8b34"
format_stamp: "Formatted at 2026-08-12 16:48:18 on dist-test-slave-1vmg"
server_key: "4da0e7a24d1c22603d187f84c23d3461"
server_key_iv: "c4623213ef464e638685ee0c5c7df071"
server_key_version: "encryptionkey@0"
I20260812 16:48:18.883682 20354 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 16:48:18.888433 20379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:18.891714 20354 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:48:18.892068 20354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal
uuid: "2d55c01b18e840ac98253279fc5e8b34"
format_stamp: "Formatted at 2026-08-12 16:48:18 on dist-test-slave-1vmg"
server_key: "4da0e7a24d1c22603d187f84c23d3461"
server_key_iv: "c4623213ef464e638685ee0c5c7df071"
server_key_version: "encryptionkey@0"
I20260812 16:48:18.892381 20354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:18.917599 20354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:18.923887 20354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:18.924289 20354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:18.946310 20354 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.61:37675
I20260812 16:48:18.946331 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.61:37675 every 8 connection(s)
I20260812 16:48:18.947938 20354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
I20260812 16:48:18.952232 20466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:18.955155 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 20354
I20260812 16:48:18.955371 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-1/wal/instance
I20260812 16:48:18.955927 29644 external_mini_cluster.cc:1428] Setting key 678acd886736084a173255aee8171e4b
I20260812 16:48:18.961711 20466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:48:18 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.60@KRBTEST.COM" created.
I20260812 16:48:18.992457 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.967100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.28.243.62:43201)
0812 16:48:18.967681 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:18.967689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:18.968809 (+ 1120us) server_negotiation.cc:407] Connection header received
0812 16:48:18.971179 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:18.971187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:18.971291 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:18.971446 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:18.973107 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.974231 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.976128 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.976804 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.976963 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:18.983697 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:18.983707 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:18.983712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:18.983788 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:18.987630 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:18.989274 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:18.989282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:18.989285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:18.989483 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:18.990221 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:18.990226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:18.990228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:18.990667 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:18.990833 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:18.991780 (+ 947us) server_negotiation.cc:299] Negotiation successful
0812 16:48:18.992043 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
I20260812 16:48:18.993388 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.967686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43201 (local address 127.0.0.1:51088)
0812 16:48:18.968606 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.968659 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:18.970823 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:18.971636 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:18.971644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:18.972208 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:18.972852 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:18.972871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.974446 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.974451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:18.975899 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:18.975917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:18.977109 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:18.977117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:18.977394 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:18.978128 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:18.978155 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:18.983105 (+ 4950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:18.987923 (+ 4818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:18.987936 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:18.987960 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:18.989044 (+ 1084us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:18.989695 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:18.989700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:18.989703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:18.989988 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:18.990857 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:18.990864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:18.991359 (+ 495us) client_negotiation.cc:769] Sending connection context
0812 16:48:18.991617 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 16:48:18.992062 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":376,"threads_started":1}
I20260812 16:48:18.995412 20466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:48:19.018455 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.996351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35780)
0812 16:48:18.996591 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:18.996612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:18.996763 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:18.998120 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:18.998126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:18.998801 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:18.999088 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:18.999103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.001468 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.001473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.002905 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.002999 (+ 94us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.004347 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.004354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.004725 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.005532 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.005567 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.009224 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.014616 (+ 5392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.014623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.014627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.015244 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.016211 (+ 967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.016215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.016218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.016389 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.017348 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.017353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.017531 (+ 178us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.017670 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.018049 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 16:48:19.018617 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:18.996626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.28.243.61:37675)
0812 16:48:18.997459 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:18.997468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:18.997504 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:48:18.997568 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:18.997577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:18.997869 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:18.998076 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:18.999293 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.001255 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.003235 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.004088 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.004218 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.009604 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.009628 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.009651 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.009706 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.014351 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.015618 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.015630 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.015633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.016018 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.016625 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.016630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.016634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.017171 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.017452 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.017841 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.018223 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":638,"thread_start_us":464,"threads_started":1}
I20260812 16:48:19.021456 20466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:48:19.023607 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.022592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60600)
0812 16:48:19.023060 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.023299 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 16:48:19.023986 20384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.026659 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.025771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60610)
0812 16:48:19.026199 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.026479 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260812 16:48:19.027163 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab.
Entry for principal kudu/127.28.243.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab.
I20260812 16:48:19.036015 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:42093
--webserver_interface=127.28.243.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab
--principal=kudu/127.28.243.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43201,127.28.243.61:37675,127.28.243.60:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:19.051424 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } attempt: 5
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:48:19.069641 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.052278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35784)
0812 16:48:19.052474 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.052496 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:19.052628 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:19.052978 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:19.052982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:19.053385 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:19.053587 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.053597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.055034 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.055039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.055969 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.055985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.057217 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.057227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.057477 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.058500 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.058528 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.061172 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.065505 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.065516 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.065521 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.066381 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.067018 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.067024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.067029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.067320 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.068065 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.068073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.068396 (+ 323us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.068683 (+ 287us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.069174 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 16:48:19.069705 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.052428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.28.243.61:37675)
0812 16:48:19.052629 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.052636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.052659 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:48:19.052754 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.052759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.052822 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.052908 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.053788 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.054779 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.056144 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.056867 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.057017 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.061669 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.061679 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.061683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.061755 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.065184 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.066653 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.066659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.066662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.066845 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.067480 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.067485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.067487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.067836 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.068056 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.069008 (+ 952us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.069428 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:48:19.071017 20342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:48:19.072628 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.071935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60622)
0812 16:48:19.072206 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.072320 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 16:48:19.073786 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.073395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60638)
0812 16:48:19.073569 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.073673 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 16:48:19.074024 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.076759 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 1
I20260812 16:48:19.078081 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.077532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60650)
0812 16:48:19.077753 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.077884 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 16:48:19.079514 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.079054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60652)
0812 16:48:19.079235 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.079357 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 16:48:19.079828 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.125363 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 1
I20260812 16:48:19.126909 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.126395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60666)
0812 16:48:19.126613 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.126720 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 16:48:19.127990 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.127640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60670)
0812 16:48:19.127791 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.127869 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 16:48:19.128422 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.148097 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 2
I20260812 16:48:19.149785 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.149170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60672)
0812 16:48:19.149430 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.149547 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 16:48:19.151592 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.151083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60676)
0812 16:48:19.151295 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.151407 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 16:48:19.151911 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.196722 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 2
I20260812 16:48:19.198387 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.197759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60690)
0812 16:48:19.198096 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.198195 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260812 16:48:19.200091 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.199593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60692)
0812 16:48:19.199834 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.199928 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260812 16:48:19.200394 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.243199 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 3
I20260812 16:48:19.244752 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.244107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60698)
0812 16:48:19.244385 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.244523 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 16:48:19.246146 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.245627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60706)
0812 16:48:19.245888 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.245996 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260812 16:48:19.246418 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.306699 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 3
I20260812 16:48:19.308529 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.307860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60722)
0812 16:48:19.308236 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.308342 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260812 16:48:19.309983 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.309543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60724)
0812 16:48:19.309760 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.309856 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 16:48:19.310403 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.389636 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 4
I20260812 16:48:19.390967 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.390362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60738)
0812 16:48:19.390614 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.390740 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 16:48:19.392194 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.391783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60740)
0812 16:48:19.391948 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.392045 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:48:19.392563 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
W20260812 16:48:19.473321 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:19.473677 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:19.473778 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:19.483428 20480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:19.483640 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:19.483697 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:19.483804 20480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:19.483881 20480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:19.487696 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 4
I20260812 16:48:19.489444 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.488758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60756)
0812 16:48:19.489136 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.489257 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260812 16:48:19.490768 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.490328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60758)
0812 16:48:19.490514 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.490605 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 16:48:19.491074 20342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:42093: Network error: Client connection negotiation failed: client connection to 127.28.243.60:42093: connect: Connection refused (error 111)
I20260812 16:48:19.497494 20480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43201,127.28.243.61:37675,127.28.243.60:42093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab
--principal=kudu/127.28.243.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:42093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:19.499611 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:19.501701 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:19.518237 20491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:19.518445 20487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:19.518445 20488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:19.519452 20480 server_base.cc:1034] running on GCE node
I20260812 16:48:19.520377 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:19.522213 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:19.523483 20480 hybrid_clock.cc:648] HybridClock initialized: now 1786553299523437 us; error 47 us; skew 500 ppm
I20260812 16:48:19.523909 20480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:19.528635 20480 init.cc:377] Logged in from keytab as kudu/127.28.243.60@KRBTEST.COM (short username kudu)
I20260812 16:48:19.530841 20480 webserver.cc:460] Webserver started at http://127.28.243.60:42379/ using document root <none> and password file <none>
I20260812 16:48:19.531741 20480 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:19.531878 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:19.532262 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:19.535708 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data/instance:
uuid: "0c4e1e9804854ab99186e68fa6c4db81"
format_stamp: "Formatted at 2026-08-12 16:48:19 on dist-test-slave-1vmg"
server_key: "6ae9d8111ba21933cf087c75455f5abf"
server_key_iv: "af4f325bb033f19f18b065aa44354b15"
server_key_version: "encryptionkey@0"
I20260812 16:48:19.536633 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal/instance:
uuid: "0c4e1e9804854ab99186e68fa6c4db81"
format_stamp: "Formatted at 2026-08-12 16:48:19 on dist-test-slave-1vmg"
server_key: "6ae9d8111ba21933cf087c75455f5abf"
server_key_iv: "af4f325bb033f19f18b065aa44354b15"
server_key_version: "encryptionkey@0"
I20260812 16:48:19.544086 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 16:48:19.550078 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:19.552165 20480 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 16:48:19.552387 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal
uuid: "0c4e1e9804854ab99186e68fa6c4db81"
format_stamp: "Formatted at 2026-08-12 16:48:19 on dist-test-slave-1vmg"
server_key: "6ae9d8111ba21933cf087c75455f5abf"
server_key_iv: "af4f325bb033f19f18b065aa44354b15"
server_key_version: "encryptionkey@0"
I20260812 16:48:19.552657 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:19.575646 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:19.581710 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:19.582105 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:19.607406 20480 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.60:42093
I20260812 16:48:19.607445 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.60:42093 every 8 connection(s)
I20260812 16:48:19.609443 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
I20260812 16:48:19.615500 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:19.618839 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 20480
I20260812 16:48:19.619159 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786553056142401-29644-0/minicluster-data/master-2/wal/instance
I20260812 16:48:19.619705 29644 external_mini_cluster.cc:1428] Setting key 40c3f23b31883319e522565f6f757095
I20260812 16:48:19.626165 20601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:19.650239 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.630650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.28.243.62:43201)
0812 16:48:19.632366 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.632371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.632389 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:48:19.632447 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.632451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.632527 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.632626 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.634512 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.635573 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.637851 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.638310 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.638439 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.643796 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.643803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.643806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.643844 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.646786 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.647857 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.647861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.647863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.648011 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.648497 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.648503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.648506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.648926 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.649090 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.649685 (+ 595us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.649895 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1606,"thread_start_us":1446,"threads_started":1}
I20260812 16:48:19.652055 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.630535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43201 (local address 127.0.0.1:51104)
0812 16:48:19.631544 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.631621 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:19.631996 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:19.632783 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:19.632791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:19.633822 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:19.634311 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.634325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.635783 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.635788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.637516 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.637579 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.638711 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.638721 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.638968 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.640279 (+ 1311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.640307 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.643525 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.646984 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.647000 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.647014 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.647653 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.648128 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.648133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.648137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.648369 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.649056 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.649063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.649284 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.649540 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.650689 (+ 1149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":420,"threads_started":1}
I20260812 16:48:19.652063 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.622464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.28.243.60:42093)
0812 16:48:19.623878 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.623905 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.623992 (+ 87us) server_negotiation.cc:407] Connection header received
0812 16:48:19.626650 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.626685 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.627408 (+ 723us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.627941 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.629710 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.631912 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.634180 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.635306 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.635508 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.642122 (+ 6614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.642139 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.642159 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.642208 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.646854 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.647796 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.647804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.647806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.648065 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.648522 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.648527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.648530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.649355 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.649625 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.650307 (+ 682us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.650833 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":232,"server-negotiator.queue_time_us":806,"spinlock_wait_cycles":17792,"thread_start_us":537,"threads_started":1}
I20260812 16:48:19.654385 20601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:19.671463 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.655335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35786)
0812 16:48:19.655512 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.655533 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:19.655656 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:19.656587 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:19.656592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:19.656987 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:19.657160 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.657169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.658316 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.658320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.659272 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.659285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.659981 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.659987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.660194 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.660778 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.660797 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.663323 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.667526 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.667531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.667534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.668103 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.668861 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.668867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.668870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.669101 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.669907 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.669912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.670317 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.670531 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.671188 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 16:48:19.671489 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.655330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.28.243.61:37675)
0812 16:48:19.656256 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.656263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.656281 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:48:19.656338 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.656343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.656408 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.656530 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.657333 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.658172 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.659414 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.659838 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.659953 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.663549 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.663555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.663558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.663608 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.667212 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.668292 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.668300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.668304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.668574 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.669254 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.669261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.669264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.669741 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.669912 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.670748 (+ 836us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.671050 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":317,"threads_started":1}
I20260812 16:48:19.673231 20601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:48:19.673712 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.656178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.28.243.62:43201)
0812 16:48:19.656344 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.656349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.656371 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:48:19.656429 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.656432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.656524 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.656635 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.658012 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.658717 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.660033 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.660445 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.660566 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.665356 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.665364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.665369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.665419 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.669626 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.671261 (+ 1635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.671269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.671273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.671498 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.672025 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.672030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.672033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.672453 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.672785 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.673112 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.673390 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:48:19.690348 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.674318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60772)
0812 16:48:19.674553 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.674578 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:19.674722 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:19.675184 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:19.675188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:19.675872 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:19.676095 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.676110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.677396 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.677403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.678664 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.678680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.679715 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.679720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.679931 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.680818 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.680841 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.684187 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.687696 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.687702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.687707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.688237 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.688637 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.688641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.688645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.688772 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.689334 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.689339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.689466 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.689604 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.689904 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 16:48:19.690369 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.674427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.28.243.60:42093)
0812 16:48:19.674647 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.674652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.674676 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:48:19.674870 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.674876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.674992 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.675124 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.676293 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.677222 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.678836 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.679505 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.679657 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.684447 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.684454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.684458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.684501 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.687534 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.688376 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.688382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.688386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.688530 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.688879 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.688883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.688885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.689212 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.689336 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.689713 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.690071 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":87}
I20260812 16:48:19.691811 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 5
I20260812 16:48:19.694480 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.676100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.28.243.61:37675)
0812 16:48:19.676283 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.676289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.676309 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:48:19.676377 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.676383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.676461 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.676573 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.678438 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.679508 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.681413 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.682352 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.682616 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.688151 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.688157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.688160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.688204 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.691399 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.692318 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.692322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.692325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.692519 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.692963 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.692968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.692972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.693290 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.693554 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.693890 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.694154 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:48:19.697182 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: Bootstrap starting.
I20260812 16:48:19.699875 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:48:19.701853 20601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:19.703322 20601 log.cc:824] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:19.707091 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: No bootstrap required, opened a new log
I20260812 16:48:19.714274 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.692503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60784)
0812 16:48:19.692695 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.692715 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:19.692817 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:19.693274 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:19.693279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:19.693795 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:19.694012 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.694023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.695658 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.695664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.696816 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.696830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.697803 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.697809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.698074 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.698781 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.698806 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.702478 (+ 3672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.708963 (+ 6485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.708973 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.708978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.709778 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.710592 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.710601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.710606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.710876 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.713102 (+ 2226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.713107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.713291 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.713461 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.713845 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 16:48:19.714257 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.692776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.28.243.60:42093)
0812 16:48:19.692939 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.692945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.692967 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:48:19.693017 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.693022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.693100 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.694170 (+ 1070us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.694310 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.695448 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.696961 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.697649 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.697780 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.702988 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.702999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.703005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.703075 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.708573 (+ 5498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.710044 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.710050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.710054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.710337 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.712313 (+ 1976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.712319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.712322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.712910 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.713104 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.713611 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.713860 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:48:19.719827 20601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.720417 20601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:19.720630 20601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4e1e9804854ab99186e68fa6c4db81, State: Initialized, Role: FOLLOWER
I20260812 16:48:19.721022 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: Bootstrap starting.
I20260812 16:48:19.721701 20601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.727363 20466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:19.728987 20630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.729512 20630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.729447 20466 log.cc:824] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:19.732409 20601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:19.738693 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: No bootstrap required, opened a new log
I20260812 16:48:19.749408 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.714346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.28.243.62:43201)
0812 16:48:19.715102 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.715108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.720154 (+ 5046us) server_negotiation.cc:407] Connection header received
0812 16:48:19.721167 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.721174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.721297 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.721398 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.728405 (+ 7007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.729548 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.735138 (+ 5590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.735835 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.735952 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.740624 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.740633 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.740638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.740696 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.745533 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.746637 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.746643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.746647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.746847 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.747435 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.747440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.747444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.747882 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.748135 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.748760 (+ 625us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.749022 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554}
I20260812 16:48:19.753749 20466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.754428 20466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:19.754616 20466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d55c01b18e840ac98253279fc5e8b34, State: Initialized, Role: FOLLOWER
I20260812 16:48:19.755210 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.713910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.28.243.61:37675)
0812 16:48:19.714130 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.714137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.720771 (+ 6634us) server_negotiation.cc:407] Connection header received
0812 16:48:19.724941 (+ 4170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.724949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.725046 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.725122 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.732183 (+ 7061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.733483 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.735406 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.736067 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.736166 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.743205 (+ 7039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.743215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.743220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.743299 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.747841 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.751897 (+ 4056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.751904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.751907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.752101 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.752572 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.752577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.752580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.753004 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.753289 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.754291 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.754696 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:48:19.755975 20466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.757330 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.757941 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.759397 20466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:19.768306 20342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } attempt: 5
I20260812 16:48:19.771216 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.720072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.28.243.60:42093)
0812 16:48:19.730276 (+ 10204us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.730283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.737470 (+ 7187us) server_negotiation.cc:407] Connection header received
0812 16:48:19.738189 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.738197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.738329 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.738409 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.748843 (+ 10434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.749946 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.752602 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.753301 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.753408 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.759999 (+ 6591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.760007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.760010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.760065 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.764699 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.766473 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.766481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.766485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.766747 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.767609 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.767614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.767617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.768087 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.770227 (+ 2140us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.770427 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.770781 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10025}
Aug 12 16:48:19 dist-test-slave-1vmg krb5kdc[20187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:48:19.767280 20480 master_runner.cc:186] Error getting master registration for 127.28.243.62:43201: OK, instance_id { permanent_uuid: "d9295f519def4323b51339765a3bc86f" instance_seqno: 1786553298325072 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:48:19.789165 20480 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 16:48:19.791124 20664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:48:19.791414 20664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:48:19.793913 20630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:48:19.794212 20630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.795948 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.771359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:42093 (local address 127.0.0.1:60810)
0812 16:48:19.771546 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:19.771565 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:19.771672 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:19.772394 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:19.772401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:19.772991 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:19.773252 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.773266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.775127 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.775134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.776780 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:19.776798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.777810 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.777817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:19.778017 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.778739 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:19.778763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:19.784890 (+ 6127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:19.792868 (+ 7978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.792875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.792878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.793546 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.794093 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:19.794099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:19.794102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:19.794306 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:19.795112 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:19.795118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:19.795261 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 16:48:19.795363 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 16:48:19.795600 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 16:48:19.796012 20630 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9295f519def4323b51339765a3bc86f (127.28.243.62:43201), 2d55c01b18e840ac98253279fc5e8b34 (127.28.243.61:37675)
I20260812 16:48:19.797123 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:19.771807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.28.243.60:42093)
0812 16:48:19.772027 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:19.772034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:19.772053 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:48:19.772110 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:19.772117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:19.772195 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:19.773428 (+ 1233us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:19.773634 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.774872 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.776994 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:19.777622 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:19.785291 (+ 7669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:19.785764 (+ 473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:19.785772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:19.785778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:19.785813 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:19.792587 (+ 6774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.793728 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.793734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.793737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.793913 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:19.794476 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:19.794481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:19.794484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:19.794941 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:19.796272 (+ 1331us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:19.796446 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 16:48:19.796749 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:48:19.797400 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c4e1e9804854ab99186e68fa6c4db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d55c01b18e840ac98253279fc5e8b34" is_pre_election: true
I20260812 16:48:19.798098 20429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4e1e9804854ab99186e68fa6c4db81 in term 0.
I20260812 16:48:19.799079 20513 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4e1e9804854ab99186e68fa6c4db81, 2d55c01b18e840ac98253279fc5e8b34; no voters:
I20260812 16:48:19.803509 20630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:48:19.803757 20630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:48:19.803865 20630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:19.807848 20630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.810204 20342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: Bootstrap starting.
I20260812 16:48:19.818076 20342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:19.819363 20630 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [CANDIDATE]: Term 1 election: Requested vote from peers d9295f519def4323b51339765a3bc86f (127.28.243.62:43201), 2d55c01b18e840ac98253279fc5e8b34 (127.28.243.61:37675)
I20260812 16:48:19.819696 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c4e1e9804854ab99186e68fa6c4db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d55c01b18e840ac98253279fc5e8b34"
I20260812 16:48:19.819978 20429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:19.820600 20342 log.cc:824] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:19.823655 20429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4e1e9804854ab99186e68fa6c4db81 in term 1.
I20260812 16:48:19.826203 20513 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4e1e9804854ab99186e68fa6c4db81, 2d55c01b18e840ac98253279fc5e8b34; no voters:
I20260812 16:48:19.826866 20630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:19.799584 20354 master_runner.cc:186] Error getting master registration for 127.28.243.62:43201: OK, instance_id { permanent_uuid: "d9295f519def4323b51339765a3bc86f" instance_seqno: 1786553298325072 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:48:19.830003 20354 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 16:48:19.833217 20630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4e1e9804854ab99186e68fa6c4db81, State: Running, Role: LEADER
I20260812 16:48:19.834018 20630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
W20260812 16:48:19.835198 20521 tablet.cc:2369] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:48:19.848080 20675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:48:19.848265 20675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:48:19.849140 20679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4e1e9804854ab99186e68fa6c4db81. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.849938 20679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:19.850879 20342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: No bootstrap required, opened a new log
I20260812 16:48:19.851425 20682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:19.853935 20682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:19.869586 20342 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.870190 20342 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:19.870338 20342 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9295f519def4323b51339765a3bc86f, State: Initialized, Role: FOLLOWER
I20260812 16:48:19.871333 20342 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } }
I20260812 16:48:19.877933 20429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [term 1 FOLLOWER]: Refusing update from remote peer 0c4e1e9804854ab99186e68fa6c4db81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:19.881425 20679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:19.883857 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.884912 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.886037 20342 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:19.887792 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c4e1e9804854ab99186e68fa6c4db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9295f519def4323b51339765a3bc86f"
I20260812 16:48:19.889318 20295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:19.887101 20298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c4e1e9804854ab99186e68fa6c4db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9295f519def4323b51339765a3bc86f" is_pre_election: true
I20260812 16:48:19.901089 20295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4e1e9804854ab99186e68fa6c4db81 in term 1.
I20260812 16:48:19.901813 20298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [term 1 FOLLOWER]: Refusing update from remote peer 0c4e1e9804854ab99186e68fa6c4db81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:19.904452 20679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:19.908841 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4e1e9804854ab99186e68fa6c4db81. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.909196 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.914115 20687 mvcc.cc:205] Tried to move back new op lower bound from 7317722316267606016 to 7317722316127281152. Current Snapshot: MvccSnapshot[applied={T|T < 7317722316267606016}]
I20260812 16:48:19.917217 20630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.917631 20630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:19.925922 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.926385 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.937557 20682 catalog_manager.cc:1347] Generated new cluster ID: 7a99797adf5146c29fc5b40126abf4de
I20260812 16:48:19.937773 20682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:19.963340 20682 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:48:19.966415 20682 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:19.967118 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4e1e9804854ab99186e68fa6c4db81. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.967481 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.973414 20630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.975087 20630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:19.976790 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c4e1e9804854ab99186e68fa6c4db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9295f519def4323b51339765a3bc86f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43201 } } peers { permanent_uuid: "2d55c01b18e840ac98253279fc5e8b34" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37675 } } peers { permanent_uuid: "0c4e1e9804854ab99186e68fa6c4db81" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 42093 } } }
I20260812 16:48:19.977084 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:48:19.977396 20698 mvcc.cc:205] Tried to move back new op lower bound from 7317722316267606016 to 7317722316127281152. Current Snapshot: MvccSnapshot[applied={T|T < 7317722316267606016}]
I20260812 16:48:20.004663 20682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c4e1e9804854ab99186e68fa6c4db81: Generated new TSK 0
I20260812 16:48:20.005964 20682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:48:20.051340 20719 catalog_manager.cc:1259] Loaded cluster ID: 7a99797adf5146c29fc5b40126abf4de
I20260812 16:48:20.051478 20719 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: loading cluster ID for follower catalog manager: success
I20260812 16:48:20.056208 20719 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: acquiring CA information for follower catalog manager: success
I20260812 16:48:20.059345 20719 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9295f519def4323b51339765a3bc86f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:48:20.850508 20675 catalog_manager.cc:1259] Loaded cluster ID: 7a99797adf5146c29fc5b40126abf4de
I20260812 16:48:20.850632 20675 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: loading cluster ID for follower catalog manager: success
I20260812 16:48:20.853615 20675 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: acquiring CA information for follower catalog manager: success
I20260812 16:48:20.855858 20675 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d55c01b18e840ac98253279fc5e8b34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:48:21.176607 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 20212
I20260812 16:48:21.179411 20332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:21.292999 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 20212
I20260812 16:48:21.326120 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 20354
I20260812 16:48:21.327972 20459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:21.460525 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 20354
I20260812 16:48:21.494124 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 20480
I20260812 16:48:21.495569 20595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:21.616416 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:21.494762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37675 (local address 127.0.0.1:35804)
0812 16:48:21.495367 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:21.495517 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":329,"threads_started":1}
W20260812 16:48:21.616868 20513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37675: connect: Connection refused (error 111)
I20260812 16:48:21.617122 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 20480
2026-08-12T16:48:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5495 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:48:21 dist-test-slave-1vmg krb5kdc[20741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:48:21 dist-test-slave-1vmg krb5kdc[20741](info): set up 2 sockets
Aug 12 16:48:21 dist-test-slave-1vmg krb5kdc[20741](info): commencing operation
krb5kdc: starting...
W20260812 16:48:23.035287 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:48:23 dist-test-slave-1vmg krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:48:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:48:23.407301 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:44965
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:23.843809 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:23.844187 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:23.844298 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:23.852749 20761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:23.852901 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:23.852973 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:23.853063 20761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:23.853137 20761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:23.865008 20761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:23.866600 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:23.868090 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:23.884414 20771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:23.884368 20773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:23.884433 20769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:23.885198 20761 server_base.cc:1034] running on GCE node
I20260812 16:48:23.886540 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:23.889253 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:23.890566 20761 hybrid_clock.cc:648] HybridClock initialized: now 1786553303890492 us; error 64 us; skew 500 ppm
I20260812 16:48:23.891083 20761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:23 dist-test-slave-1vmg krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553303, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:23.895105 20761 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:23.896900 20761 webserver.cc:460] Webserver started at http://127.28.243.62:43203/ using document root <none> and password file <none>
I20260812 16:48:23.897722 20761 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:23.897869 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:23.898239 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:23.902016 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "c31c260a645b420ba24ff599aa1061f9"
format_stamp: "Formatted at 2026-08-12 16:48:23 on dist-test-slave-1vmg"
server_key: "54d6f1185a0ae776c5fff23f2798beb2"
server_key_iv: "722bec3dcd841d2ffd5c4668404c6e29"
server_key_version: "encryptionkey@0"
I20260812 16:48:23.903223 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "c31c260a645b420ba24ff599aa1061f9"
format_stamp: "Formatted at 2026-08-12 16:48:23 on dist-test-slave-1vmg"
server_key: "54d6f1185a0ae776c5fff23f2798beb2"
server_key_iv: "722bec3dcd841d2ffd5c4668404c6e29"
server_key_version: "encryptionkey@0"
I20260812 16:48:23.909514 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:48:23.913995 20789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:23.915822 20761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:48:23.916033 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "c31c260a645b420ba24ff599aa1061f9"
format_stamp: "Formatted at 2026-08-12 16:48:23 on dist-test-slave-1vmg"
server_key: "54d6f1185a0ae776c5fff23f2798beb2"
server_key_iv: "722bec3dcd841d2ffd5c4668404c6e29"
server_key_version: "encryptionkey@0"
I20260812 16:48:23.916254 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:23.931588 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:23.936892 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:23.937214 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:23.961449 20761 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:44965
I20260812 16:48:23.961468 20884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:44965 every 8 connection(s)
I20260812 16:48:23.963146 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:23.967499 20885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:23.968500 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 20761
I20260812 16:48:23.968768 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:48:23.969594 29644 external_mini_cluster.cc:1428] Setting key 7efcdb327020cd5cefd5d8150db29498
I20260812 16:48:23.985122 20885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9: Bootstrap starting.
I20260812 16:48:23.988585 20885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9: Neither blocks nor log segments found. Creating new log.
Aug 12 16:48:23 dist-test-slave-1vmg krb5kdc[20741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:23.990428 20885 log.cc:824] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:23.994565 20885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9: No bootstrap required, opened a new log
I20260812 16:48:24.001443 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:23.972214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.28.243.62:44965)
0812 16:48:23.973466 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:23.973486 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:23.973557 (+ 71us) server_negotiation.cc:407] Connection header received
0812 16:48:23.975843 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:23.975887 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:23.976605 (+ 718us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:23.977180 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:23.978855 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:23.981176 (+ 2321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:23.983344 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:23.984521 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:23.984791 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:23.991483 (+ 6692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:23.991493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:23.991515 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:23.991575 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:23.995602 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:23.996522 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:23.996532 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:23.996544 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:23.996786 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:23.997305 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:23.997310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:23.997313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:23.998176 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:23.998529 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:23.999235 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 16:48:23.999811 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":628,"threads_started":1}
I20260812 16:48:24.005640 20885 raft_consensus.cc:348] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31c260a645b420ba24ff599aa1061f9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44965 } }
I20260812 16:48:24.006068 20885 raft_consensus.cc:374] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:24.006196 20885 raft_consensus.cc:729] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31c260a645b420ba24ff599aa1061f9, State: Initialized, Role: FOLLOWER
I20260812 16:48:24.006834 20885 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31c260a645b420ba24ff599aa1061f9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44965 } }
I20260812 16:48:24.007157 20885 raft_consensus.cc:388] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:24.007304 20885 raft_consensus.cc:482] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:24.007498 20885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:24.007879 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:48:24.009806 20885 raft_consensus.cc:504] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31c260a645b420ba24ff599aa1061f9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44965 } }
I20260812 16:48:24.010350 20885 leader_election.cc:302] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c31c260a645b420ba24ff599aa1061f9; no voters:
I20260812 16:48:24.011139 20885 leader_election.cc:288] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:48:24.011210 20896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:24.012166 20896 raft_consensus.cc:686] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [term 1 LEADER]: Becoming Leader. State: Replica: c31c260a645b420ba24ff599aa1061f9, State: Running, Role: LEADER
I20260812 16:48:24.012691 20896 consensus_queue.cc:237] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31c260a645b420ba24ff599aa1061f9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44965 } }
I20260812 16:48:24.013370 20885 sys_catalog.cc:564] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:24.016291 20897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c31c260a645b420ba24ff599aa1061f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31c260a645b420ba24ff599aa1061f9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44965 } } }
I20260812 16:48:24.016264 20898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31c260a645b420ba24ff599aa1061f9. Latest consensus state: current_term: 1 leader_uuid: "c31c260a645b420ba24ff599aa1061f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31c260a645b420ba24ff599aa1061f9" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 44965 } } }
I20260812 16:48:24.016781 20898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:24.016788 20897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:24.018604 20904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:24.025143 20904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:24.037117 20904 catalog_manager.cc:1347] Generated new cluster ID: b6d4584ca296446683ca5c2b81aae96f
I20260812 16:48:24.037266 20904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:24.079885 20904 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:48:24.081532 20904 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:24.091171 20904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c31c260a645b420ba24ff599aa1061f9: Generated new TSK 0
I20260812 16:48:24.092167 20904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:48:24.126802 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 20761
I20260812 16:48:24.128604 20875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:24.232036 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 20761
2026-08-12T16:48:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2610 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:48:24 dist-test-slave-1vmg krb5kdc[20938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:48:24 dist-test-slave-1vmg krb5kdc[20938](info): set up 2 sockets
Aug 12 16:48:24 dist-test-slave-1vmg krb5kdc[20938](info): commencing operation
krb5kdc: starting...
W20260812 16:48:25.599453 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:48:25 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:48:25.964279 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40897
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:26.417991 20954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:26.418345 20954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:26.418444 20954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:26.427132 20954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:26.427306 20954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:26.427369 20954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:26.427435 20954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:26.427488 20954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:26.439587 20954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:26.441164 20954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:26.442601 20954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:26.456902 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:26.456923 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:26.457314 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:26.457809 20954 server_base.cc:1034] running on GCE node
I20260812 16:48:26.458731 20954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:26.461206 20954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:26.462561 20954 hybrid_clock.cc:648] HybridClock initialized: now 1786553306462461 us; error 90 us; skew 500 ppm
I20260812 16:48:26.463074 20954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:26 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553306, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:26.466612 20954 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:26.468911 20954 webserver.cc:460] Webserver started at http://127.28.243.62:38469/ using document root <none> and password file <none>
I20260812 16:48:26.469728 20954 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:26.469868 20954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:26.470211 20954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:26.473744 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "3195de405bf74af98602ecf31e8556f1"
format_stamp: "Formatted at 2026-08-12 16:48:26 on dist-test-slave-1vmg"
server_key: "ff98f63a8d5c96f0f92caf953557a922"
server_key_iv: "6c28efbc62a189aa957e085c20ccd416"
server_key_version: "encryptionkey@0"
I20260812 16:48:26.474627 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "3195de405bf74af98602ecf31e8556f1"
format_stamp: "Formatted at 2026-08-12 16:48:26 on dist-test-slave-1vmg"
server_key: "ff98f63a8d5c96f0f92caf953557a922"
server_key_iv: "6c28efbc62a189aa957e085c20ccd416"
server_key_version: "encryptionkey@0"
I20260812 16:48:26.481226 20954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260812 16:48:26.487743 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:26.489880 20954 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260812 16:48:26.490109 20954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "3195de405bf74af98602ecf31e8556f1"
format_stamp: "Formatted at 2026-08-12 16:48:26 on dist-test-slave-1vmg"
server_key: "ff98f63a8d5c96f0f92caf953557a922"
server_key_iv: "6c28efbc62a189aa957e085c20ccd416"
server_key_version: "encryptionkey@0"
I20260812 16:48:26.490347 20954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:26.526372 20954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:26.532856 20954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:26.533179 20954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:26.554849 20954 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:40897
I20260812 16:48:26.554881 21083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:40897 every 8 connection(s)
I20260812 16:48:26.556397 20954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:26.560577 21088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:26.563239 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 20954
I20260812 16:48:26.563476 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:48:26.564414 29644 external_mini_cluster.cc:1428] Setting key d5b2dc10a776bcdad30685bf1f7d8308
I20260812 16:48:26.577075 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap starting.
I20260812 16:48:26.581171 21088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:26.582434 21088 log.cc:824] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:48:26 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:26.587541 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: No bootstrap required, opened a new log
I20260812 16:48:26.597885 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:26.568306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.28.243.62:40897)
0812 16:48:26.569492 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:26.569515 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:26.569589 (+ 74us) server_negotiation.cc:407] Connection header received
0812 16:48:26.572192 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:26.572233 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:26.572856 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:26.573435 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:26.575587 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:26.577597 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:26.579345 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:26.580468 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:26.580688 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:26.587071 (+ 6383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:26.587092 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:26.587120 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:26.587181 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:26.591963 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:26.593008 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:26.593020 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:26.593030 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:26.593303 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:26.593745 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:26.593750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:26.593754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:26.594729 (+ 975us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:26.595134 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:26.595724 (+ 590us) server_negotiation.cc:299] Negotiation successful
0812 16:48:26.596264 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":486,"threads_started":1}
I20260812 16:48:26.598858 21088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:26.599411 21088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:26.599531 21088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3195de405bf74af98602ecf31e8556f1, State: Initialized, Role: FOLLOWER
I20260812 16:48:26.600224 21088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:26.600466 21088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:26.600625 21088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:26.600804 21088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:26.604550 21088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:26.605664 21088 leader_election.cc:302] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3195de405bf74af98602ecf31e8556f1; no voters:
I20260812 16:48:26.606626 21088 leader_election.cc:288] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:48:26.606791 21099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:26.609091 21099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3195de405bf74af98602ecf31e8556f1, State: Running, Role: LEADER
I20260812 16:48:26.609884 21099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:26.610262 21088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:26.612890 21101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3195de405bf74af98602ecf31e8556f1. Latest consensus state: current_term: 1 leader_uuid: "3195de405bf74af98602ecf31e8556f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } } }
I20260812 16:48:26.613018 21100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3195de405bf74af98602ecf31e8556f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } } }
I20260812 16:48:26.613402 21101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:26.615756 21100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:26.615981 21109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:26.621945 21109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:26.636377 21109 catalog_manager.cc:1347] Generated new cluster ID: 8aeeade267524bfd9e0e98dd00fce30b
I20260812 16:48:26.636524 21109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:26.653738 21109 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:48:26.655380 21109 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:26.665549 21109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Generated new TSK 0
I20260812 16:48:26.666874 21109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:48:26.754596 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40897
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:27.205684 21128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:27.206234 21128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:27.206573 21128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:27.215397 21128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:27.215559 21128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:27.215713 21128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:48:27.227411 21128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:27.229050 21128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:27.230520 21128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:27.245707 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:27.245749 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:27.246272 21139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:27.246253 21128 server_base.cc:1034] running on GCE node
I20260812 16:48:27.247139 21128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:27.248216 21128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:27.249486 21128 hybrid_clock.cc:648] HybridClock initialized: now 1786553307249438 us; error 41 us; skew 500 ppm
I20260812 16:48:27.249863 21128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:27 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553307, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:27.259896 21128 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:48:27.261617 21128 webserver.cc:460] Webserver started at http://127.28.243.1:37643/ using document root <none> and password file <none>
I20260812 16:48:27.262372 21128 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:27.262506 21128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:27.262836 21128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:27.266083 21128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "c3767dcce4084e5d92d473b9cdee32a6"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "3fadf483badcfe5e02d86f3e94b577e7"
server_key_iv: "acebc2005acc226c8c218d15598cc4e1"
server_key_version: "encryptionkey@0"
I20260812 16:48:27.268019 21128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "c3767dcce4084e5d92d473b9cdee32a6"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "3fadf483badcfe5e02d86f3e94b577e7"
server_key_iv: "acebc2005acc226c8c218d15598cc4e1"
server_key_version: "encryptionkey@0"
I20260812 16:48:27.274443 21128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 16:48:27.278448 21156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:27.280088 21128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:48:27.280498 21128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "c3767dcce4084e5d92d473b9cdee32a6"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "3fadf483badcfe5e02d86f3e94b577e7"
server_key_iv: "acebc2005acc226c8c218d15598cc4e1"
server_key_version: "encryptionkey@0"
I20260812 16:48:27.280742 21128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:27.299358 21128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:27.304646 21128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:27.304930 21128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:27.305953 21128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:27.307483 21128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:27.307564 21128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:27.307663 21128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:27.307710 21128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:27.349646 21128 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:44217
I20260812 16:48:27.349682 21351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:44217 every 8 connection(s)
I20260812 16:48:27.351204 21128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:48:27.360733 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 21128
I20260812 16:48:27.360954 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:48:27.361601 29644 external_mini_cluster.cc:1428] Setting key 1587dea990f6d47428f24514be9f5dcd
Aug 12 16:48:27 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553307, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:27.378618 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:27.354061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:41361 (local address 127.28.243.62:40897)
0812 16:48:27.354584 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:27.354591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:27.355662 (+ 1071us) server_negotiation.cc:407] Connection header received
0812 16:48:27.357594 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:27.357599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:27.357675 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:27.357782 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:27.359635 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.360530 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.363886 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.364598 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.364764 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:27.370581 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:27.370589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:27.370593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:27.370641 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:27.374414 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:27.375734 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:27.375740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:27.375743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:27.375942 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:27.376475 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:27.376481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:27.376484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:27.376853 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:27.377008 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:27.377943 (+ 935us) server_negotiation.cc:299] Negotiation successful
0812 16:48:27.378235 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
I20260812 16:48:27.379886 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:27.354632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:41361)
0812 16:48:27.355490 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:27.355532 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:27.357273 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:27.358155 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:27.358164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:27.358738 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:27.359433 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:27.359452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.360736 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.360742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:27.363685 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:27.363708 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.364887 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.364893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:27.365536 (+ 643us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:27.366637 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:27.366668 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:27.370225 (+ 3557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:27.374904 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:27.374951 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:27.374975 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:27.375565 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:27.376101 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:27.376105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:27.376108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:27.376349 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:27.377079 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:27.377086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:27.377488 (+ 402us) client_negotiation.cc:769] Sending connection context
0812 16:48:27.377791 (+ 303us) client_negotiation.cc:240] Negotiation successful
0812 16:48:27.378397 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":347,"threads_started":1}
I20260812 16:48:27.382261 21352 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40897
I20260812 16:48:27.382660 21352 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:27.384227 21352 heartbeater.cc:511] Master 127.28.243.62:40897 requested a full tablet report, sending...
I20260812 16:48:27.390103 21021 ts_manager.cc:194] Registered new tserver with Master: c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1:44217)
I20260812 16:48:27.392557 21021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:41361
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:48:27.439751 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40897
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:27.828814 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:27.829898 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:27.830090 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:27.837944 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:27.838114 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:27.838246 21359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:48:27.849464 21359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:27.851207 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:27.852718 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:27.868465 21369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:27.868439 21368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:27.868497 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:27.869206 21359 server_base.cc:1034] running on GCE node
I20260812 16:48:27.869916 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:27.871083 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:27.872309 21359 hybrid_clock.cc:648] HybridClock initialized: now 1786553307872263 us; error 39 us; skew 500 ppm
I20260812 16:48:27.872665 21359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:27 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553307, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:27.876320 21359 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:48:27.878320 21359 webserver.cc:460] Webserver started at http://127.28.243.2:40197/ using document root <none> and password file <none>
I20260812 16:48:27.879268 21359 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:27.879437 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:27.879789 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:27.883581 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "638b4337670844982fdf71b1ae25dfad"
server_key_iv: "1ff34971b9ab2b3bab2eec2611955c0c"
server_key_version: "encryptionkey@0"
I20260812 16:48:27.884898 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "638b4337670844982fdf71b1ae25dfad"
server_key_iv: "1ff34971b9ab2b3bab2eec2611955c0c"
server_key_version: "encryptionkey@0"
I20260812 16:48:27.892120 21359 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260812 16:48:27.896785 21388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:27.898635 21359 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:48:27.898854 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "638b4337670844982fdf71b1ae25dfad"
server_key_iv: "1ff34971b9ab2b3bab2eec2611955c0c"
server_key_version: "encryptionkey@0"
I20260812 16:48:27.900442 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:27.923259 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:27.930459 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:27.930788 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:27.932030 21359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:27.933856 21359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:27.933951 21359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:27.934051 21359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:27.934116 21359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:27.974390 21359 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:44735
I20260812 16:48:27.974433 21612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:44735 every 8 connection(s)
I20260812 16:48:27.975826 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:48:27.977142 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 21359
I20260812 16:48:27.977331 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:48:27.977864 29644 external_mini_cluster.cc:1428] Setting key 49a1691d4d226eb205f55b9b840ff587
Aug 12 16:48:27 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553307, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:28.005036 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:27.979294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:34569 (local address 127.28.243.62:40897)
0812 16:48:27.979772 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:27.979776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:27.981514 (+ 1738us) server_negotiation.cc:407] Connection header received
0812 16:48:27.984991 (+ 3477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:27.984997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:27.985081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:27.985235 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:27.988020 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.988798 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.991216 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.991664 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.991797 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:27.996581 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:27.996588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:27.996591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:27.996630 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:27.999617 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:28.001345 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:28.001352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:28.001355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:28.001645 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:28.002626 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:28.002633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:28.002636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:28.003137 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:28.003351 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.004273 (+ 922us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.004568 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260812 16:48:28.007063 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:27.979715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:34569)
0812 16:48:27.981237 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:27.981339 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:27.984597 (+ 3258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:27.985520 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:27.985532 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:27.986781 (+ 1249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:27.987767 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:27.987799 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.988990 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.988996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:27.990994 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:27.991014 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:27.991989 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:27.991998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:27.992317 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:27.993318 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:27.993350 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:27.996263 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:27.999906 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:27.999917 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:27.999933 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:28.000456 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:28.001816 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:28.001821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:28.001824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:28.002359 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:28.003327 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:28.003339 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:28.003794 (+ 455us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.004124 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.004776 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":581,"threads_started":1}
I20260812 16:48:28.009509 21613 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40897
I20260812 16:48:28.009900 21613 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:28.010810 21613 heartbeater.cc:511] Master 127.28.243.62:40897 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
I20260812 16:48:28.013017 21021 ts_manager.cc:194] Registered new tserver with Master: 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735)
I20260812 16:48:28.014002 21021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:34569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:48:28.054790 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40897
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:28.396654 21352 heartbeater.cc:503] Master 127.28.243.62:40897 was elected leader, sending a full tablet report...
W20260812 16:48:28.436369 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:28.436750 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:28.436894 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:28.445068 21628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:28.445243 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:28.445407 21628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:48:28.456522 21628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:28.457963 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:28.459474 21628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:28.473738 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:28.473707 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:28.474113 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:28.474761 21628 server_base.cc:1034] running on GCE node
I20260812 16:48:28.475493 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:28.476574 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:28.477828 21628 hybrid_clock.cc:648] HybridClock initialized: now 1786553308477754 us; error 57 us; skew 500 ppm
I20260812 16:48:28.478230 21628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:28 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:28.481941 21628 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:48:28.483877 21628 webserver.cc:460] Webserver started at http://127.28.243.3:39645/ using document root <none> and password file <none>
I20260812 16:48:28.484724 21628 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:28.484855 21628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:28.485165 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:28.489805 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "49e45c009a3b4a23a48a7a4e24c1f03c"
format_stamp: "Formatted at 2026-08-12 16:48:28 on dist-test-slave-1vmg"
server_key: "84e072b8e7df2429f56ca2339a435a46"
server_key_iv: "e2237be204fc45f1a77bfa525ed4c463"
server_key_version: "encryptionkey@0"
I20260812 16:48:28.490731 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "49e45c009a3b4a23a48a7a4e24c1f03c"
format_stamp: "Formatted at 2026-08-12 16:48:28 on dist-test-slave-1vmg"
server_key: "84e072b8e7df2429f56ca2339a435a46"
server_key_iv: "e2237be204fc45f1a77bfa525ed4c463"
server_key_version: "encryptionkey@0"
I20260812 16:48:28.496878 21628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 16:48:28.502105 21654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:28.504874 21628 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 16:48:28.505196 21628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "49e45c009a3b4a23a48a7a4e24c1f03c"
format_stamp: "Formatted at 2026-08-12 16:48:28 on dist-test-slave-1vmg"
server_key: "84e072b8e7df2429f56ca2339a435a46"
server_key_iv: "e2237be204fc45f1a77bfa525ed4c463"
server_key_version: "encryptionkey@0"
I20260812 16:48:28.505890 21628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:28.562661 21628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:28.569845 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:28.570315 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:28.571802 21628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:28.573452 21628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:28.573549 21628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:28.573652 21628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:28.573696 21628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:28.615269 21628 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:36151
I20260812 16:48:28.615316 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:36151 every 8 connection(s)
I20260812 16:48:28.616691 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:48:28.622148 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 21628
I20260812 16:48:28.622354 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:48:28.622790 29644 external_mini_cluster.cc:1428] Setting key aeca5892cdf50e03df468819b069706c
Aug 12 16:48:28 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:28.641929 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.619458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:45145 (local address 127.28.243.62:40897)
0812 16:48:28.619938 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.619943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.620919 (+ 976us) server_negotiation.cc:407] Connection header received
0812 16:48:28.623270 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.623277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.623378 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.623500 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:28.626242 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.627183 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.629677 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.630109 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.630230 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.634198 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:28.634206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:28.634209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:28.634252 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:28.637601 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:28.638617 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:28.638624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:28.638627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:28.638798 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:28.639713 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:28.639717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:28.639721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:28.640114 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:28.640283 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.641411 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.641633 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260812 16:48:28.643751 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.619998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:45145)
0812 16:48:28.620753 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:28.620795 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:28.622688 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:28.623972 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:28.623986 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:28.624979 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:28.625968 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.625999 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.627449 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.627457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.629458 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.629486 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.630395 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.630402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.630642 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.631420 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:28.631452 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:28.633872 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:28.637816 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:28.637826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:28.637844 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:28.638493 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:28.638990 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:28.638996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:28.639000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:28.639327 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:28.640281 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:28.640290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:28.640850 (+ 560us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.641262 (+ 412us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.641770 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":281,"threads_started":1}
I20260812 16:48:28.645594 21866 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40897
I20260812 16:48:28.645925 21866 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:28.646735 21866 heartbeater.cc:511] Master 127.28.243.62:40897 requested a full tablet report, sending...
I20260812 16:48:28.648509 21021 ts_manager.cc:194] Registered new tserver with Master: 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3:36151)
I20260812 16:48:28.649415 21021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:45145
I20260812 16:48:28.653322 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:48:28.675665 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.658435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.28.243.62:40897)
0812 16:48:28.658712 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.658719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.659198 (+ 479us) server_negotiation.cc:407] Connection header received
0812 16:48:28.659418 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.659422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.659513 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.659605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:28.661497 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.662388 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.664006 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.664435 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.664639 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.669315 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:28.669326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:28.669330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:28.669376 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:28.672060 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:28.673209 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:28.673215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:28.673217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:28.673408 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:28.673922 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:28.673926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:28.673929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:28.674356 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:28.674568 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.675128 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.675337 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 16:48:28.682075 21021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:48:28.692155 21021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:48:28.724352 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.710599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44217 (local address 127.0.0.1:50820)
0812 16:48:28.712390 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:28.712411 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:28.712532 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:28.714712 (+ 2180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:28.714717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:28.714747 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:28.715484 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.715497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.717395 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.717400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.720717 (+ 3317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.720740 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.722963 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.722972 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.723563 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.723608 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.723712 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.723888 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"mutex_wait_us":357,"thread_start_us":4131,"threads_started":1}
I20260812 16:48:28.724587 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.709903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:36151 (local address 127.0.0.1:38150)
0812 16:48:28.711290 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:28.711330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:28.711732 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:28.714434 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:28.714445 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:28.714490 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:28.715221 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.715237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.717316 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.717322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.720717 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.720740 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.722898 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.722941 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.723674 (+ 733us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.723709 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.723921 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.724090 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":202,"thread_start_us":861,"threads_started":1}
I20260812 16:48:28.724956 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.710669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.28.243.3:36151)
0812 16:48:28.713758 (+ 3089us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.713765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.713801 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:48:28.713892 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.713898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.714130 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.714366 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:28.715494 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.717170 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.720978 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.722589 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.722759 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.723053 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.724010 (+ 957us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.724198 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2889,"thread_start_us":2826,"threads_started":1}
I20260812 16:48:28.726181 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.711716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:44735 (local address 127.0.0.1:49790)
0812 16:48:28.715137 (+ 3421us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:28.715167 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:28.715408 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:28.716209 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:28.716213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:28.716240 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:28.716629 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.716641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.718290 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.718296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.721011 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.721038 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.725327 (+ 4289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.725335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.725515 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.725550 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.725641 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.725797 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3254,"mutex_wait_us":3,"thread_start_us":689,"threads_started":1}
I20260812 16:48:28.726627 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.711456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.28.243.1:44217)
0812 16:48:28.713677 (+ 2221us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.713685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.713719 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:48:28.713837 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.713846 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.714146 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.714366 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:28.715732 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.717200 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.720977 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.722588 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.725578 (+ 2990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.725947 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.726027 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.726220 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2038,"thread_start_us":1851,"threads_started":1}
I20260812 16:48:28.727156 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.712235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.28.243.2:44735)
0812 16:48:28.713109 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.713118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.715280 (+ 2162us) server_negotiation.cc:407] Connection header received
0812 16:48:28.715635 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.715644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.715910 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.716103 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:28.716818 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.718138 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.721273 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.725066 (+ 3793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.725262 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.725476 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.726574 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.726779 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":587,"threads_started":1}
I20260812 16:48:28.729269 21745 tablet_service.cc:1467] Processing CreateTablet for tablet 591aeab8647a42f19d970983e7cd21c7 (DEFAULT_TABLE table=test-table [id=f583996a454c410a9bacca812cb7af05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:28.731364 21745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591aeab8647a42f19d970983e7cd21c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:28.731312 21245 tablet_service.cc:1467] Processing CreateTablet for tablet 591aeab8647a42f19d970983e7cd21c7 (DEFAULT_TABLE table=test-table [id=f583996a454c410a9bacca812cb7af05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:28.731743 21485 tablet_service.cc:1467] Processing CreateTablet for tablet 591aeab8647a42f19d970983e7cd21c7 (DEFAULT_TABLE table=test-table [id=f583996a454c410a9bacca812cb7af05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:28.732913 21245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591aeab8647a42f19d970983e7cd21c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:28.732926 21485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591aeab8647a42f19d970983e7cd21c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:28.746014 21902 tablet_bootstrap.cc:492] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Bootstrap starting.
I20260812 16:48:28.749569 21902 tablet_bootstrap.cc:654] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:28.750696 21908 tablet_bootstrap.cc:492] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Bootstrap starting.
I20260812 16:48:28.751232 21902 log.cc:824] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:28.755766 21906 tablet_bootstrap.cc:492] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Bootstrap starting.
I20260812 16:48:28.759357 21908 tablet_bootstrap.cc:654] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:28.760120 21906 tablet_bootstrap.cc:654] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:28.760878 21908 log.cc:824] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:28.761353 21902 tablet_bootstrap.cc:492] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: No bootstrap required, opened a new log
I20260812 16:48:28.761404 21906 log.cc:824] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:28.761655 21902 ts_tablet_manager.cc:1397] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260812 16:48:28.764755 21906 tablet_bootstrap.cc:492] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: No bootstrap required, opened a new log
I20260812 16:48:28.765054 21908 tablet_bootstrap.cc:492] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: No bootstrap required, opened a new log
I20260812 16:48:28.765107 21906 ts_tablet_manager.cc:1397] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260812 16:48:28.765470 21908 ts_tablet_manager.cc:1397] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260812 16:48:28.773113 21906 raft_consensus.cc:348] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.773571 21906 raft_consensus.cc:374] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:28.773672 21906 raft_consensus.cc:729] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e45c009a3b4a23a48a7a4e24c1f03c, State: Initialized, Role: FOLLOWER
I20260812 16:48:28.774379 21902 raft_consensus.cc:348] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.774770 21906 consensus_queue.cc:260] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.775195 21902 raft_consensus.cc:374] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:28.775326 21902 raft_consensus.cc:729] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c102cfc2f5a4f1aa83ef11c6e761ce4, State: Initialized, Role: FOLLOWER
I20260812 16:48:28.776371 21902 consensus_queue.cc:260] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.777551 21866 heartbeater.cc:503] Master 127.28.243.62:40897 was elected leader, sending a full tablet report...
I20260812 16:48:28.778285 21906 ts_tablet_manager.cc:1428] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260812 16:48:28.779186 21908 raft_consensus.cc:348] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.779750 21908 raft_consensus.cc:374] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:28.779870 21908 raft_consensus.cc:729] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3767dcce4084e5d92d473b9cdee32a6, State: Initialized, Role: FOLLOWER
I20260812 16:48:28.780751 21908 consensus_queue.cc:260] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.781366 21613 heartbeater.cc:503] Master 127.28.243.62:40897 was elected leader, sending a full tablet report...
I20260812 16:48:28.782177 21902 ts_tablet_manager.cc:1428] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Time spent starting tablet: real 0.020s user 0.021s sys 0.000s
I20260812 16:48:28.784101 21908 ts_tablet_manager.cc:1428] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Time spent starting tablet: real 0.018s user 0.008s sys 0.009s
W20260812 16:48:28.854738 21353 tablet.cc:2369] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:48:28.868476 21867 tablet.cc:2369] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:28.894804 21916 raft_consensus.cc:482] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:48:28.895188 21916 raft_consensus.cc:504] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.897822 21916 leader_election.cc:288] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1:44217), 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735)
I20260812 16:48:28.905731 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.898762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:44583 (local address 127.28.243.1:44217)
0812 16:48:28.899045 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.899051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.899077 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:48:28.899158 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.899217 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.899329 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.899451 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:28.899961 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.900908 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.903159 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.904726 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.904869 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.904935 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.905396 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.905528 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 16:48:28.905791 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.898278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44217 (local address 127.28.243.3:44583)
0812 16:48:28.898722 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:28.898750 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:28.898952 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:28.899521 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:28.899526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:28.899559 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:28.899794 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.899807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.901100 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.901106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.902808 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.902830 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.904917 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.904925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.905146 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.905185 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.905301 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.905459 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":30}
I20260812 16:48:28.907080 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.898774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:60719 (local address 127.28.243.2:44735)
0812 16:48:28.899045 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.899052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.899698 (+ 646us) server_negotiation.cc:407] Connection header received
0812 16:48:28.899834 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.899839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.899924 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.900000 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:28.900749 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.901775 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.904733 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.906058 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.906252 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.906327 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.906679 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.906800 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:48:28.907356 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.898348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:44735 (local address 127.28.243.3:60719)
0812 16:48:28.899579 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:28.899595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:28.899711 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:28.900051 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:28.900056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:28.900105 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:28.900449 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.900461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.902106 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.902113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.904469 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:28.904492 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.906307 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.906313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:28.906495 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.906529 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:48:28.906626 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 16:48:28.906793 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":4,"thread_start_us":878,"threads_started":1}
I20260812 16:48:28.907337 21281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591aeab8647a42f19d970983e7cd21c7" candidate_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3767dcce4084e5d92d473b9cdee32a6" is_pre_election: true
I20260812 16:48:28.907954 21281 raft_consensus.cc:2454] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e45c009a3b4a23a48a7a4e24c1f03c in term 0.
I20260812 16:48:28.908515 21517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591aeab8647a42f19d970983e7cd21c7" candidate_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" is_pre_election: true
I20260812 16:48:28.908748 21662 leader_election.cc:302] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e45c009a3b4a23a48a7a4e24c1f03c, c3767dcce4084e5d92d473b9cdee32a6; no voters:
I20260812 16:48:28.909403 21916 raft_consensus.cc:2781] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:48:28.909410 21517 raft_consensus.cc:2454] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e45c009a3b4a23a48a7a4e24c1f03c in term 0.
I20260812 16:48:28.909844 21916 raft_consensus.cc:482] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:48:28.909991 21916 raft_consensus.cc:3037] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:28.913360 21916 raft_consensus.cc:504] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.914116 21916 leader_election.cc:288] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [CANDIDATE]: Term 1 election: Requested vote from peers c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1:44217), 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735)
I20260812 16:48:28.915102 21281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591aeab8647a42f19d970983e7cd21c7" candidate_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3767dcce4084e5d92d473b9cdee32a6"
I20260812 16:48:28.915403 21281 raft_consensus.cc:3037] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:28.918455 21281 raft_consensus.cc:2454] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e45c009a3b4a23a48a7a4e24c1f03c in term 1.
I20260812 16:48:28.919441 21662 leader_election.cc:302] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e45c009a3b4a23a48a7a4e24c1f03c, c3767dcce4084e5d92d473b9cdee32a6; no voters:
I20260812 16:48:28.919258 21517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591aeab8647a42f19d970983e7cd21c7" candidate_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
I20260812 16:48:28.919685 21517 raft_consensus.cc:3037] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:28.919855 21916 raft_consensus.cc:2781] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:28.920481 21916 raft_consensus.cc:686] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 LEADER]: Becoming Leader. State: Replica: 49e45c009a3b4a23a48a7a4e24c1f03c, State: Running, Role: LEADER
I20260812 16:48:28.921021 21916 consensus_queue.cc:237] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } }
I20260812 16:48:28.922674 21517 raft_consensus.cc:2454] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e45c009a3b4a23a48a7a4e24c1f03c in term 1.
I20260812 16:48:28.926299 21021 catalog_manager.cc:5495] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3). New cstate: current_term: 1 leader_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } health_report { overall_health: UNKNOWN } } }
I20260812 16:48:28.970300 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:28.964547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.28.243.3:36151)
0812 16:48:28.964786 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:28.964792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:28.964814 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:48:28.965024 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:28.965028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:28.965114 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:28.965271 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:48:28.965654 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.966375 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.967988 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:28.968406 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:28.968527 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:28.969120 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:48:28.969653 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:48:28.969819 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:28.969923 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 16:48:28.970016 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 16:48:28.979022 21614 tablet.cc:2369] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:28.979559 21517 raft_consensus.cc:1261] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Refusing update from remote peer 49e45c009a3b4a23a48a7a4e24c1f03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:28.979565 21281 raft_consensus.cc:1261] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Refusing update from remote peer 49e45c009a3b4a23a48a7a4e24c1f03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:28.980685 21921 consensus_queue.cc:1035] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:28.981058 21916 consensus_queue.cc:1035] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:29.017735 21021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:29.018100 21021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:29.022746 21021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Sending DeleteTablet for 3 replicas of tablet 591aeab8647a42f19d970983e7cd21c7
I20260812 16:48:29.024313 21245 tablet_service.cc:1514] Processing DeleteTablet for tablet 591aeab8647a42f19d970983e7cd21c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:29 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:50820
I20260812 16:48:29.024315 21745 tablet_service.cc:1514] Processing DeleteTablet for tablet 591aeab8647a42f19d970983e7cd21c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:29 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:38150
I20260812 16:48:29.024567 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 21128
I20260812 16:48:29.025956 21485 tablet_service.cc:1514] Processing DeleteTablet for tablet 591aeab8647a42f19d970983e7cd21c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:29 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:49790
I20260812 16:48:29.034766 21955 tablet_replica.cc:330] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: stopping tablet replica
I20260812 16:48:29.035413 21955 raft_consensus.cc:2229] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:29.036068 21955 raft_consensus.cc:2258] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:29.041757 21949 tablet_replica.cc:330] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: stopping tablet replica
I20260812 16:48:29.041996 21955 ts_tablet_manager.cc:1905] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:29.042650 21949 raft_consensus.cc:2229] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:29.043402 21949 raft_consensus.cc:2258] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:29.046638 21949 ts_tablet_manager.cc:1905] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:29.050060 21955 ts_tablet_manager.cc:1918] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:29.050308 21955 log.cc:1196] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/591aeab8647a42f19d970983e7cd21c7
I20260812 16:48:29.051043 21952 tablet_replica.cc:330] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: stopping tablet replica
I20260812 16:48:29.051674 21952 raft_consensus.cc:2229] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:48:29.053031 21952 raft_consensus.cc:2258] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:29.056318 21952 ts_tablet_manager.cc:1905] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:29.056365 21949 ts_tablet_manager.cc:1918] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:29.056550 21949 log.cc:1196] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/591aeab8647a42f19d970983e7cd21c7
I20260812 16:48:29.057606 21949 ts_tablet_manager.cc:1939] T 591aeab8647a42f19d970983e7cd21c7 P c3767dcce4084e5d92d473b9cdee32a6: Deleting consensus metadata
I20260812 16:48:29.059952 21955 ts_tablet_manager.cc:1939] T 591aeab8647a42f19d970983e7cd21c7 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Deleting consensus metadata
I20260812 16:48:29.059932 20990 catalog_manager.cc:4841] TS c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1:44217): tablet 591aeab8647a42f19d970983e7cd21c7 (table test-table [id=f583996a454c410a9bacca812cb7af05]) successfully deleted
I20260812 16:48:29.062117 20989 catalog_manager.cc:4841] TS 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735): tablet 591aeab8647a42f19d970983e7cd21c7 (table test-table [id=f583996a454c410a9bacca812cb7af05]) successfully deleted
Aug 12 16:48:29 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:48:29.065001 21952 ts_tablet_manager.cc:1918] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:29.065241 21952 log.cc:1196] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/591aeab8647a42f19d970983e7cd21c7
I20260812 16:48:29.065868 21952 ts_tablet_manager.cc:1939] T 591aeab8647a42f19d970983e7cd21c7 P 49e45c009a3b4a23a48a7a4e24c1f03c: Deleting consensus metadata
I20260812 16:48:29.067685 20989 catalog_manager.cc:4841] TS 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3:36151): tablet 591aeab8647a42f19d970983e7cd21c7 (table test-table [id=f583996a454c410a9bacca812cb7af05]) successfully deleted
I20260812 16:48:29.076603 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:29.025932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.28.243.1:44217)
0812 16:48:29.026695 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:29.026703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:29.039228 (+ 12525us) server_negotiation.cc:407] Connection header received
0812 16:48:29.051407 (+ 12179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:29.051414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:29.051510 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:29.051598 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:29.053778 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:29.054996 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:29.057958 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:29.058646 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:29.058784 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:29.065793 (+ 7009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:29.065817 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:29.065840 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:29.065897 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:29.071084 (+ 5187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:29.072676 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:29.072688 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:29.072692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:29.072915 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:29.073466 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:29.073472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:29.073475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:29.074004 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:29.074385 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:29.075251 (+ 866us) server_negotiation.cc:299] Negotiation successful
0812 16:48:29.075669 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
I20260812 16:48:29.078227 21344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:29.172364 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 21128
I20260812 16:48:29.199014 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 21359
Aug 12 16:48:29 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:48:29.223717 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:29.199863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.28.243.2:44735)
0812 16:48:29.200033 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:29.200038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:29.200058 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:48:29.200312 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:29.200317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:29.200404 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:29.200494 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:29.202373 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:29.203265 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:29.205281 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:29.205679 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:29.205806 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:29.216145 (+ 10339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:29.216165 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:29.216183 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:29.216234 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:29.220526 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:29.221385 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:29.221396 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:29.221399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:29.221539 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:29.221925 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:29.221931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:29.221935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:29.222395 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:29.222689 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:29.223080 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 16:48:29.223340 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:48:29.224512 21605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:29.316926 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 21359
I20260812 16:48:29.344434 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 21628
Aug 12 16:48:29 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:48:29.371059 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:29.345364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.28.243.3:36151)
0812 16:48:29.345572 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:29.345578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:29.345602 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:48:29.345790 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:29.345795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:29.345877 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:29.346009 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:29.348182 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:29.349019 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:29.352841 (+ 3822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:29.353552 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:29.353695 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:29.361463 (+ 7768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:29.361492 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:29.361517 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:29.361608 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:29.366082 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:29.367303 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:29.367313 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:29.367317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:29.367555 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:29.368116 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:29.368121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:29.368124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:29.368654 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:29.369156 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:29.369936 (+ 780us) server_negotiation.cc:299] Negotiation successful
0812 16:48:29.370355 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:48:29.371903 21855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:29.507014 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 21628
I20260812 16:48:29.540931 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 20954
I20260812 16:48:29.542335 21077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:29.656423 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 20954
I20260812 16:48:29.687167 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40897
--webserver_interface=127.28.243.62
--webserver_port=38469
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:30.075911 21964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:30.076264 21964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:30.076361 21964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:30.084189 21964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:30.084350 21964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:30.084400 21964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:30.084452 21964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:30.084498 21964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:30.096427 21964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:30.098124 21964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:30.099669 21964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:30.131553 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:30.131635 21982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:30.131592 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:30.132535 21964 server_base.cc:1034] running on GCE node
I20260812 16:48:30.133514 21964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:30.135442 21964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:30.136734 21964 hybrid_clock.cc:648] HybridClock initialized: now 1786553310136642 us; error 66 us; skew 500 ppm
I20260812 16:48:30.137763 21964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:30 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:30.142155 21964 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:30.145921 21964 webserver.cc:460] Webserver started at http://127.28.243.62:38469/ using document root <none> and password file <none>
I20260812 16:48:30.146961 21964 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:30.147153 21964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:30.154556 21964 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 16:48:30.158316 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:30.160000 21964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 16:48:30.160233 21964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "3195de405bf74af98602ecf31e8556f1"
format_stamp: "Formatted at 2026-08-12 16:48:26 on dist-test-slave-1vmg"
server_key: "ff98f63a8d5c96f0f92caf953557a922"
server_key_iv: "6c28efbc62a189aa957e085c20ccd416"
server_key_version: "encryptionkey@0"
I20260812 16:48:30.161438 21964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:30.187342 21964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:30.192710 21964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:30.193153 21964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:30.215968 21964 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:40897
I20260812 16:48:30.215926 22085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:40897 every 8 connection(s)
I20260812 16:48:30.217715 21964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:30.221475 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 21964
I20260812 16:48:30.222685 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:44217
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:40897
--webserver_port=37643
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:30.229539 22086 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:48:30.234984 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap starting.
I20260812 16:48:30.259788 22086 log.cc:824] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:30.274307 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:48:30.274902 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap complete.
I20260812 16:48:30.286405 22086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:30.287523 22086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3195de405bf74af98602ecf31e8556f1, State: Initialized, Role: FOLLOWER
I20260812 16:48:30.288311 22086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:30.288575 22086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:30.288710 22086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:30.288894 22086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:48:30.293298 22086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:30.293949 22086 leader_election.cc:302] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3195de405bf74af98602ecf31e8556f1; no voters:
I20260812 16:48:30.294616 22086 leader_election.cc:288] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:48:30.294880 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:48:30.296245 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 LEADER]: Becoming Leader. State: Replica: 3195de405bf74af98602ecf31e8556f1, State: Running, Role: LEADER
I20260812 16:48:30.296900 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:30.297286 22086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:30.299682 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3195de405bf74af98602ecf31e8556f1. Latest consensus state: current_term: 2 leader_uuid: "3195de405bf74af98602ecf31e8556f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } } }
I20260812 16:48:30.300313 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:30.300848 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3195de405bf74af98602ecf31e8556f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } } }
I20260812 16:48:30.301256 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:30.303668 22109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:30.315100 22109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:30.317052 22109 catalog_manager.cc:1259] Loaded cluster ID: 8aeeade267524bfd9e0e98dd00fce30b
I20260812 16:48:30.317215 22109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:30.321835 22109 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 16:48:30.342139 22109 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 16:48:30.353832 22109 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:30.356068 22109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Loaded TSK: 0
I20260812 16:48:30.357834 22109 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:48:30.636619 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:30.636981 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:30.637122 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:30.644805 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:30.644949 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:30.645087 22091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:48:30.656038 22091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:44217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=37643
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:30.657531 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:30.658983 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:30.675570 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:30.675784 22141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:30.675987 22149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:30.676232 22091 server_base.cc:1034] running on GCE node
I20260812 16:48:30.677241 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:30.678666 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:30.679967 22091 hybrid_clock.cc:648] HybridClock initialized: now 1786553310679868 us; error 86 us; skew 500 ppm
I20260812 16:48:30.680413 22091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:30 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:30.688619 22091 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:48:30.690512 22091 webserver.cc:460] Webserver started at http://127.28.243.1:37643/ using document root <none> and password file <none>
I20260812 16:48:30.691380 22091 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:30.691525 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:30.697677 22091 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 16:48:30.701097 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:30.702823 22091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:48:30.703064 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "c3767dcce4084e5d92d473b9cdee32a6"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "3fadf483badcfe5e02d86f3e94b577e7"
server_key_iv: "acebc2005acc226c8c218d15598cc4e1"
server_key_version: "encryptionkey@0"
I20260812 16:48:30.704136 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:30.729264 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:30.735754 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:30.736109 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:30.739437 22091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:30.741503 22091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:30.741624 22091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:30.741734 22091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:30.741823 22091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:30.789994 22091 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:44217
I20260812 16:48:30.790042 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:44217 every 8 connection(s)
I20260812 16:48:30.792593 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:48:30.800755 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 22091
I20260812 16:48:30.801527 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:44735
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:40897
--webserver_port=40197
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:48:30 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:30.824841 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:30.796743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:44147)
0812 16:48:30.797769 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:30.797825 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:30.800352 (+ 2527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:30.802343 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:30.802357 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:30.803285 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:30.804222 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:30.804254 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:30.806732 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:30.806738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:30.808271 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:30.808291 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:30.810202 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:30.810210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:30.810547 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:30.811760 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:30.811802 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:30.814733 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:30.819992 (+ 5259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:30.820001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:30.820016 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:30.820557 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:30.821177 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:30.821182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:30.821186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:30.821419 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:30.822410 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:30.822418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:30.822875 (+ 457us) client_negotiation.cc:769] Sending connection context
0812 16:48:30.823222 (+ 347us) client_negotiation.cc:240] Negotiation successful
0812 16:48:30.823706 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":642,"threads_started":1}
I20260812 16:48:30.825152 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:30.797804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:44147 (local address 127.28.243.62:40897)
0812 16:48:30.798444 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:30.798455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:30.798502 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:48:30.801084 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:30.801104 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:30.801640 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:30.802105 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:30.804585 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:30.806450 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:30.808467 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:30.809719 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:30.809964 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:30.815301 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:30.815314 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:30.815337 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:30.815395 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:30.819673 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:30.820746 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:30.820756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:30.820764 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:30.821022 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:30.821563 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:30.821568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:30.821571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:30.822260 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:30.822457 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:30.823493 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0812 16:48:30.823981 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":271,"threads_started":1}
I20260812 16:48:30.828406 22374 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40897
I20260812 16:48:30.828743 22374 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:30.829546 22374 heartbeater.cc:511] Master 127.28.243.62:40897 requested a full tablet report, sending...
I20260812 16:48:30.832294 22023 ts_manager.cc:194] Registered new tserver with Master: c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1:44217)
I20260812 16:48:30.834503 22023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:44147
W20260812 16:48:31.196437 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:31.196758 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:31.196859 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:31.205782 22382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:31.205929 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:31.206049 22382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:48:31.218257 22382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:44735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=40197
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:31.219782 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:31.221375 22382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:31.236923 22396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:31.236966 22391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:31.236862 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:31.238011 22382 server_base.cc:1034] running on GCE node
I20260812 16:48:31.239127 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:31.240262 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:31.241506 22382 hybrid_clock.cc:648] HybridClock initialized: now 1786553311241454 us; error 50 us; skew 500 ppm
I20260812 16:48:31.241871 22382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:31 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:31.245317 22382 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:48:31.247175 22382 webserver.cc:460] Webserver started at http://127.28.243.2:40197/ using document root <none> and password file <none>
I20260812 16:48:31.247941 22382 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:31.248075 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:31.255471 22382 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:48:31.259338 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:31.261059 22382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:48:31.261251 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "638b4337670844982fdf71b1ae25dfad"
server_key_iv: "1ff34971b9ab2b3bab2eec2611955c0c"
server_key_version: "encryptionkey@0"
I20260812 16:48:31.262324 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:31.283062 22382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:31.288139 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:31.288434 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:31.289990 22382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:31.291986 22382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:31.292131 22382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:31.292246 22382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:31.292330 22382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:31.330168 22382 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:44735
I20260812 16:48:31.330142 22617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:44735 every 8 connection(s)
I20260812 16:48:31.332904 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:48:31.340660 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 22382
I20260812 16:48:31.341305 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:36151
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:40897
--webserver_port=39645
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:48:31 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:31.357740 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.336447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:57803 (local address 127.28.243.62:40897)
0812 16:48:31.337115 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.337120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.338083 (+ 963us) server_negotiation.cc:407] Connection header received
0812 16:48:31.340344 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.340351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.340458 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.340561 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:31.342449 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.343237 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.344946 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.345407 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.345527 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.349938 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:31.349944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:31.349947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:31.349987 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:31.353095 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:31.354498 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:31.354505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:31.354508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:31.354713 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:31.355453 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:31.355457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:31.355460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:31.355851 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:31.356012 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.357128 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.357452 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260812 16:48:31.359006 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.337066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:57803)
0812 16:48:31.337898 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:31.337947 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:31.339992 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:31.340897 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:31.340908 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:31.341565 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:31.342229 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.342250 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.343422 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.343427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.344766 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.344784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.345668 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.345676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.345957 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.346673 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:31.346705 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:31.349626 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:31.353342 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:31.353351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:31.353368 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:31.354315 (+ 947us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:31.354844 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:31.354850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:31.354854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:31.355165 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:31.356010 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:31.356019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:31.356564 (+ 545us) client_negotiation.cc:769] Sending connection context
0812 16:48:31.356906 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 16:48:31.357494 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":326,"threads_started":1}
I20260812 16:48:31.361052 22619 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40897
I20260812 16:48:31.361456 22619 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:31.362457 22619 heartbeater.cc:511] Master 127.28.243.62:40897 requested a full tablet report, sending...
I20260812 16:48:31.364604 22023 ts_manager.cc:194] Registered new tserver with Master: 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735)
I20260812 16:48:31.365760 22023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:57803
W20260812 16:48:31.735204 22628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:31.735532 22628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:31.735666 22628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:31.743674 22628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:31.743825 22628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:31.743978 22628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:48:31.754946 22628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:36151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=39645
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:31.756557 22628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:31.758095 22628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:31.775249 22634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:31.775358 22639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:31.775327 22635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:31.776341 22628 server_base.cc:1034] running on GCE node
I20260812 16:48:31.777202 22628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:31.778404 22628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:31.779698 22628 hybrid_clock.cc:648] HybridClock initialized: now 1786553311779539 us; error 157 us; skew 500 ppm
I20260812 16:48:31.780092 22628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:31 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:31.783736 22628 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:48:31.785512 22628 webserver.cc:460] Webserver started at http://127.28.243.3:39645/ using document root <none> and password file <none>
I20260812 16:48:31.786306 22628 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:31.786458 22628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:31.793771 22628 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:48:31.797674 22650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:31.799381 22628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:48:31.799564 22628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "49e45c009a3b4a23a48a7a4e24c1f03c"
format_stamp: "Formatted at 2026-08-12 16:48:28 on dist-test-slave-1vmg"
server_key: "84e072b8e7df2429f56ca2339a435a46"
server_key_iv: "e2237be204fc45f1a77bfa525ed4c463"
server_key_version: "encryptionkey@0"
I20260812 16:48:31.800673 22628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:31.823662 22628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:31.829547 22628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:31.829845 22628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:31.831113 22628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:31.832772 22628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:31.832871 22628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:31.832984 22628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:31.833048 22628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:31.839011 22374 heartbeater.cc:503] Master 127.28.243.62:40897 was elected leader, sending a full tablet report...
I20260812 16:48:31.872224 22628 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:36151
I20260812 16:48:31.873812 22628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:48:31.875495 22851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:36151 every 8 connection(s)
I20260812 16:48:31.881948 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 22628
Aug 12 16:48:31 dist-test-slave-1vmg krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553311, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:31.908079 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.883462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.28.243.62:40897)
0812 16:48:31.886456 (+ 2994us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.886464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.886492 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:48:31.886575 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.886582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.886690 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.886968 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:31.888814 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.890039 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.891776 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.892429 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.892587 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.896815 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:31.896821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:31.896824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:31.896865 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:31.901426 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:31.902681 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:31.902690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:31.902694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:31.903544 (+ 850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:31.904334 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:31.904341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:31.904345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:31.905147 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:31.905845 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.906060 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.906905 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2800,"thread_start_us":2583,"threads_started":1}
I20260812 16:48:31.911689 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.882158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:46145 (local address 127.28.243.62:40897)
0812 16:48:31.883143 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.883150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.884606 (+ 1456us) server_negotiation.cc:407] Connection header received
0812 16:48:31.887128 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.887133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.887208 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.887279 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:31.890433 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.891282 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.893470 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.894062 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.894185 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.899185 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:31.899193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:31.899198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:31.899265 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:31.906035 (+ 6770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:31.907732 (+ 1697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:31.907740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:31.907744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:31.907999 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:31.908861 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:31.908869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:31.908872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:31.909556 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:31.909749 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.910859 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.911246 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":765,"threads_started":1}
I20260812 16:48:31.912726 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.879447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:46145)
0812 16:48:31.882891 (+ 3444us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:31.883045 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:31.886635 (+ 3590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:31.888218 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:31.888231 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:31.889155 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:31.890223 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.890256 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.891475 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.891480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.893294 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.893312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.894366 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.894375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.894617 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.895475 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:31.895514 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:31.898812 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:31.906259 (+ 7447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:31.906272 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:31.906290 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:31.907013 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:31.908257 (+ 1244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:31.908263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:31.908268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:31.908626 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:31.909716 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:31.909739 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:31.910344 (+ 605us) client_negotiation.cc:769] Sending connection context
0812 16:48:31.910711 (+ 367us) client_negotiation.cc:240] Negotiation successful
0812 16:48:31.911380 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2631,"thread_start_us":2174,"threads_started":1}
I20260812 16:48:31.914732 22853 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40897
I20260812 16:48:31.915135 22853 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:31.915990 22853 heartbeater.cc:511] Master 127.28.243.62:40897 requested a full tablet report, sending...
I20260812 16:48:31.918054 22023 ts_manager.cc:194] Registered new tserver with Master: 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3:36151)
I20260812 16:48:31.919615 22023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:46145
I20260812 16:48:31.926347 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:48:31.937071 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.929204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.28.243.62:40897)
0812 16:48:31.929495 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.929502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.929924 (+ 422us) server_negotiation.cc:407] Connection header received
0812 16:48:31.930233 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.930238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.930587 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.930813 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:48:31.931506 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.932300 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.934059 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.934631 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.934803 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.935686 (+ 883us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:48:31.936338 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:48:31.936487 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.936660 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.936753 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 16:48:31.938280 22023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:48:31.947124 22023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:48:31.976958 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.964002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.28.243.1:44217)
0812 16:48:31.966643 (+ 2641us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.966653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.966679 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:48:31.966764 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.966771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.967098 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.967686 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:31.969009 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.970961 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.973790 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.975345 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.975542 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.975838 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.976356 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.976563 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2378,"thread_start_us":2134,"threads_started":1}
I20260812 16:48:31.977375 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.963416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44217 (local address 127.0.0.1:50840)
0812 16:48:31.964768 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:31.964818 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:31.965025 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:31.967364 (+ 2339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:31.967370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:31.967413 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:31.968324 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.968422 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.971168 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.971175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.973613 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.973630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.975695 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.975705 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.975982 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.976068 (+ 86us) client_negotiation.cc:769] Sending connection context
0812 16:48:31.976254 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 16:48:31.976678 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":2,"thread_start_us":907,"threads_started":1}
I20260812 16:48:31.977401 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.963182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.28.243.2:44735)
0812 16:48:31.963885 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.963896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.966106 (+ 2210us) server_negotiation.cc:407] Connection header received
0812 16:48:31.966561 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.966570 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.966877 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.967113 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:31.969268 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.970689 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.974299 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.975854 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.976009 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.976190 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.976603 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.976800 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":703,"threads_started":1}
I20260812 16:48:31.978844 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.963523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:44735 (local address 127.0.0.1:49800)
0812 16:48:31.965974 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:31.965995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:31.966334 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:31.967500 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:31.967504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:31.967530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:31.969025 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.969045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.970842 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.970847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.973977 (+ 3130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.974012 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.976158 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.976165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.976360 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.976401 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 16:48:31.977323 (+ 922us) client_negotiation.cc:240] Negotiation successful
0812 16:48:31.977496 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2044,"mutex_wait_us":1,"thread_start_us":1975,"threads_started":1}
I20260812 16:48:31.979909 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.967764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:36151 (local address 127.0.0.1:38186)
0812 16:48:31.970015 (+ 2251us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:31.970037 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:31.970168 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:31.970881 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:31.970886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:31.970943 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:31.971506 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.971520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.973632 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.973637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.975891 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:31.975908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.979031 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.979039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:31.979245 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.979278 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:48:31.979359 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:48:31.979513 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2116,"mutex_wait_us":32,"thread_start_us":1952,"threads_started":1}
I20260812 16:48:31.980016 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:31.967161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.28.243.3:36151)
0812 16:48:31.968251 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:31.968444 (+ 193us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:31.970156 (+ 1712us) server_negotiation.cc:407] Connection header received
0812 16:48:31.970336 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:31.970345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:31.970618 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:31.970855 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:31.971768 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.973480 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.976227 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:31.978735 (+ 2508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:31.978883 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:31.979165 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:31.979464 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 16:48:31.979647 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":675,"threads_started":1}
I20260812 16:48:31.982995 22259 tablet_service.cc:1467] Processing CreateTablet for tablet 0f47fe5997b044cc97c0abb7d5868b80 (DEFAULT_TABLE table=test-table [id=e0b33aaaee844f02b5553c1839fa112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:31.983299 22502 tablet_service.cc:1467] Processing CreateTablet for tablet 0f47fe5997b044cc97c0abb7d5868b80 (DEFAULT_TABLE table=test-table [id=e0b33aaaee844f02b5553c1839fa112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:31.984787 22502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f47fe5997b044cc97c0abb7d5868b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:31.984354 22748 tablet_service.cc:1467] Processing CreateTablet for tablet 0f47fe5997b044cc97c0abb7d5868b80 (DEFAULT_TABLE table=test-table [id=e0b33aaaee844f02b5553c1839fa112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:31.985030 22259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f47fe5997b044cc97c0abb7d5868b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:31.985958 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f47fe5997b044cc97c0abb7d5868b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:31.998798 22881 tablet_bootstrap.cc:492] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Bootstrap starting.
I20260812 16:48:32.002564 22881 tablet_bootstrap.cc:654] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:32.003305 22882 tablet_bootstrap.cc:492] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Bootstrap starting.
I20260812 16:48:32.004274 22881 log.cc:824] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:32.007056 22883 tablet_bootstrap.cc:492] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Bootstrap starting.
I20260812 16:48:32.008010 22882 tablet_bootstrap.cc:654] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:32.009701 22882 log.cc:824] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:32.011317 22881 tablet_bootstrap.cc:492] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: No bootstrap required, opened a new log
I20260812 16:48:32.011677 22881 ts_tablet_manager.cc:1397] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260812 16:48:32.013005 22883 tablet_bootstrap.cc:654] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:32.013754 22882 tablet_bootstrap.cc:492] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: No bootstrap required, opened a new log
I20260812 16:48:32.014070 22882 ts_tablet_manager.cc:1397] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260812 16:48:32.014271 22883 log.cc:824] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:32.017460 22883 tablet_bootstrap.cc:492] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: No bootstrap required, opened a new log
I20260812 16:48:32.017813 22883 ts_tablet_manager.cc:1397] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260812 16:48:32.025784 22883 raft_consensus.cc:348] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.026266 22883 raft_consensus.cc:374] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:32.026374 22883 raft_consensus.cc:729] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e45c009a3b4a23a48a7a4e24c1f03c, State: Initialized, Role: FOLLOWER
I20260812 16:48:32.027088 22883 consensus_queue.cc:260] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.027002 22881 raft_consensus.cc:348] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.028594 22853 heartbeater.cc:503] Master 127.28.243.62:40897 was elected leader, sending a full tablet report...
I20260812 16:48:32.028820 22881 raft_consensus.cc:374] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:32.028967 22881 raft_consensus.cc:729] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c102cfc2f5a4f1aa83ef11c6e761ce4, State: Initialized, Role: FOLLOWER
I20260812 16:48:32.029551 22883 ts_tablet_manager.cc:1428] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260812 16:48:32.030062 22881 consensus_queue.cc:260] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.030587 22882 raft_consensus.cc:348] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.031266 22882 raft_consensus.cc:374] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:32.031373 22882 raft_consensus.cc:729] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3767dcce4084e5d92d473b9cdee32a6, State: Initialized, Role: FOLLOWER
I20260812 16:48:32.032406 22882 consensus_queue.cc:260] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.033141 22619 heartbeater.cc:503] Master 127.28.243.62:40897 was elected leader, sending a full tablet report...
I20260812 16:48:32.034566 22881 ts_tablet_manager.cc:1428] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Time spent starting tablet: real 0.023s user 0.011s sys 0.010s
I20260812 16:48:32.035812 22882 ts_tablet_manager.cc:1428] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Time spent starting tablet: real 0.022s user 0.023s sys 0.000s
I20260812 16:48:32.037070 22895 raft_consensus.cc:482] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:48:32.037865 22895 raft_consensus.cc:504] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.040953 22895 leader_election.cc:288] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735), 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3:36151)
W20260812 16:48:32.045773 22375 tablet.cc:2369] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:32.050350 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.041680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:39045 (local address 127.28.243.2:44735)
0812 16:48:32.042177 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:32.042182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:32.042536 (+ 354us) server_negotiation.cc:407] Connection header received
0812 16:48:32.042645 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:32.042651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:32.042731 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:32.042836 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:32.043922 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.045439 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.047980 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.048869 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.049038 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.049124 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:32.049756 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 16:48:32.050064 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 16:48:32.050395 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.041324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:44735 (local address 127.28.243.1:39045)
0812 16:48:32.042150 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:32.042173 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:32.042369 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:32.043115 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:32.043120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:32.043154 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:32.043651 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:32.043665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.045712 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.045726 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:32.047765 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:32.047781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.049275 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.049290 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:32.049498 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.049535 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:48:32.049667 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 16:48:32.049861 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":413,"threads_started":1}
I20260812 16:48:32.051760 22539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f47fe5997b044cc97c0abb7d5868b80" candidate_uuid: "c3767dcce4084e5d92d473b9cdee32a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" is_pre_election: true
I20260812 16:48:32.052404 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.043038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:49829 (local address 127.28.243.3:36151)
0812 16:48:32.043484 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:32.043490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:32.044461 (+ 971us) server_negotiation.cc:407] Connection header received
0812 16:48:32.044618 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:32.044623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:32.044692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:32.044992 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:32.045800 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.047303 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.050032 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.051159 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.051391 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.051475 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:32.051974 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 16:48:32.052137 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:48:32.052461 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.042395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:36151 (local address 127.28.243.1:49829)
0812 16:48:32.044344 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:32.044361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:32.044491 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:32.045006 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:32.045012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:32.045061 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:32.045582 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:32.045596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.047509 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.047513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:32.049793 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:32.049810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.051532 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.051538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:32.051706 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.051741 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:48:32.051872 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 16:48:32.052013 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"thread_start_us":1611,"threads_started":1}
I20260812 16:48:32.052424 22539 raft_consensus.cc:2454] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3767dcce4084e5d92d473b9cdee32a6 in term 0.
I20260812 16:48:32.053272 22172 leader_election.cc:302] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c102cfc2f5a4f1aa83ef11c6e761ce4, c3767dcce4084e5d92d473b9cdee32a6; no voters:
I20260812 16:48:32.053697 22895 raft_consensus.cc:2781] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:48:32.053833 22895 raft_consensus.cc:482] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:48:32.053915 22895 raft_consensus.cc:3037] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:32.053830 22772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f47fe5997b044cc97c0abb7d5868b80" candidate_uuid: "c3767dcce4084e5d92d473b9cdee32a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" is_pre_election: true
I20260812 16:48:32.054383 22772 raft_consensus.cc:2454] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3767dcce4084e5d92d473b9cdee32a6 in term 0.
I20260812 16:48:32.056717 22895 raft_consensus.cc:504] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.057471 22895 leader_election.cc:288] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [CANDIDATE]: Term 1 election: Requested vote from peers 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735), 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3:36151)
I20260812 16:48:32.058123 22772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f47fe5997b044cc97c0abb7d5868b80" candidate_uuid: "c3767dcce4084e5d92d473b9cdee32a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c"
I20260812 16:48:32.058146 22539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f47fe5997b044cc97c0abb7d5868b80" candidate_uuid: "c3767dcce4084e5d92d473b9cdee32a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
I20260812 16:48:32.058347 22772 raft_consensus.cc:3037] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:32.058380 22539 raft_consensus.cc:3037] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:32.061115 22772 raft_consensus.cc:2454] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3767dcce4084e5d92d473b9cdee32a6 in term 1.
I20260812 16:48:32.061729 22172 leader_election.cc:302] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e45c009a3b4a23a48a7a4e24c1f03c, c3767dcce4084e5d92d473b9cdee32a6; no voters:
I20260812 16:48:32.062222 22895 raft_consensus.cc:2781] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:32.062338 22539 raft_consensus.cc:2454] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3767dcce4084e5d92d473b9cdee32a6 in term 1.
I20260812 16:48:32.063182 22895 raft_consensus.cc:686] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 LEADER]: Becoming Leader. State: Replica: c3767dcce4084e5d92d473b9cdee32a6, State: Running, Role: LEADER
I20260812 16:48:32.064083 22895 consensus_queue.cc:237] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } }
I20260812 16:48:32.072219 22021 catalog_manager.cc:5495] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1). New cstate: current_term: 1 leader_uuid: "c3767dcce4084e5d92d473b9cdee32a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3767dcce4084e5d92d473b9cdee32a6" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44217 } health_report { overall_health: HEALTHY } } }
W20260812 16:48:32.085947 22621 tablet.cc:2369] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:32.112285 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.104564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.28.243.1:44217)
0812 16:48:32.104795 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:32.104801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:32.104826 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:48:32.105365 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:32.105370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:32.105479 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:32.105694 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:48:32.106527 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.107453 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.109592 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.110065 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.110198 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.110959 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:48:32.111568 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:48:32.111696 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:32.111902 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 16:48:32.111995 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:48:32.122993 22772 raft_consensus.cc:1261] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Refusing update from remote peer c3767dcce4084e5d92d473b9cdee32a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:32.123713 22539 raft_consensus.cc:1261] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Refusing update from remote peer c3767dcce4084e5d92d473b9cdee32a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:32.124469 22895 consensus_queue.cc:1035] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e45c009a3b4a23a48a7a4e24c1f03c" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:32.125154 22902 consensus_queue.cc:1035] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:48:32.128715 22854 tablet.cc:2369] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:32.138041 22908 mvcc.cc:205] Tried to move back new op lower bound from 7317722366438453248 to 7317722366222073856. Current Snapshot: MvccSnapshot[applied={T|T < 7317722366438453248}]
I20260812 16:48:32.138484 22909 mvcc.cc:205] Tried to move back new op lower bound from 7317722366438453248 to 7317722366222073856. Current Snapshot: MvccSnapshot[applied={T|T < 7317722366438453248}]
I20260812 16:48:32.157264 22021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:32.157570 22021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:32.162369 22021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Sending DeleteTablet for 3 replicas of tablet 0f47fe5997b044cc97c0abb7d5868b80
I20260812 16:48:32.164095 22502 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f47fe5997b044cc97c0abb7d5868b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:32 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:49800
I20260812 16:48:32.164610 22259 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f47fe5997b044cc97c0abb7d5868b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:32 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:50840
I20260812 16:48:32.164661 22748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f47fe5997b044cc97c0abb7d5868b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:32 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:38186
I20260812 16:48:32.171996 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 22091
I20260812 16:48:32.179352 22927 tablet_replica.cc:330] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: stopping tablet replica
I20260812 16:48:32.179586 22921 tablet_replica.cc:330] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: stopping tablet replica
I20260812 16:48:32.179991 22927 raft_consensus.cc:2229] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:48:32.180176 22921 raft_consensus.cc:2229] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:32.180775 22921 raft_consensus.cc:2258] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:32.181107 22927 raft_consensus.cc:2258] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:32.181949 22925 tablet_replica.cc:330] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: stopping tablet replica
I20260812 16:48:32.182471 22925 raft_consensus.cc:2229] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:32.183271 22925 raft_consensus.cc:2258] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:32.183336 22921 ts_tablet_manager.cc:1905] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:32.185743 22925 ts_tablet_manager.cc:1905] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:32.185761 22927 ts_tablet_manager.cc:1905] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:32.194399 22925 ts_tablet_manager.cc:1918] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:32.194734 22925 log.cc:1196] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/0f47fe5997b044cc97c0abb7d5868b80
I20260812 16:48:32.194865 22921 ts_tablet_manager.cc:1918] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:32.195164 22921 log.cc:1196] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/0f47fe5997b044cc97c0abb7d5868b80
I20260812 16:48:32.195580 22925 ts_tablet_manager.cc:1939] T 0f47fe5997b044cc97c0abb7d5868b80 P 49e45c009a3b4a23a48a7a4e24c1f03c: Deleting consensus metadata
I20260812 16:48:32.195770 22921 ts_tablet_manager.cc:1939] T 0f47fe5997b044cc97c0abb7d5868b80 P 2c102cfc2f5a4f1aa83ef11c6e761ce4: Deleting consensus metadata
I20260812 16:48:32.198185 22927 ts_tablet_manager.cc:1918] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:32.198313 22000 catalog_manager.cc:4841] TS 2c102cfc2f5a4f1aa83ef11c6e761ce4 (127.28.243.2:44735): tablet 0f47fe5997b044cc97c0abb7d5868b80 (table test-table [id=e0b33aaaee844f02b5553c1839fa112a]) successfully deleted
I20260812 16:48:32.198571 22927 log.cc:1196] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/0f47fe5997b044cc97c0abb7d5868b80
I20260812 16:48:32.198710 22000 catalog_manager.cc:4841] TS 49e45c009a3b4a23a48a7a4e24c1f03c (127.28.243.3:36151): tablet 0f47fe5997b044cc97c0abb7d5868b80 (table test-table [id=e0b33aaaee844f02b5553c1839fa112a]) successfully deleted
I20260812 16:48:32.199308 22927 ts_tablet_manager.cc:1939] T 0f47fe5997b044cc97c0abb7d5868b80 P c3767dcce4084e5d92d473b9cdee32a6: Deleting consensus metadata
I20260812 16:48:32.201265 22001 catalog_manager.cc:4841] TS c3767dcce4084e5d92d473b9cdee32a6 (127.28.243.1:44217): tablet 0f47fe5997b044cc97c0abb7d5868b80 (table test-table [id=e0b33aaaee844f02b5553c1839fa112a]) successfully deleted
I20260812 16:48:32.205336 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.173230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.28.243.1:44217)
0812 16:48:32.173813 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:32.173820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:32.173841 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:48:32.173917 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:32.173923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:32.174029 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:32.174132 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:32.176412 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.177814 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.180132 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.180923 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.181061 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.185666 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:32.185683 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:32.185703 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:32.185771 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:32.200677 (+ 14906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:32.202187 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:32.202202 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:32.202206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:32.202491 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:32.203318 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:32.203324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:32.203327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:32.203860 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:32.204215 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:32.204610 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 16:48:32.204908 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260812 16:48:32.206586 22367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:32.310424 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 22091
I20260812 16:48:32.344046 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 22382
I20260812 16:48:32.365453 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.344938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.28.243.2:44735)
0812 16:48:32.345152 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:32.345158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:32.345280 (+ 122us) server_negotiation.cc:407] Connection header received
0812 16:48:32.345483 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:32.345489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:32.345570 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:32.345668 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:32.347214 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.348046 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.349449 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.349856 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.350125 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.358627 (+ 8502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:32.358647 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:32.358666 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:32.358726 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:32.361938 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:32.362860 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:32.362869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:32.362872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:32.363078 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:32.363622 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:32.363628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:32.363631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:32.364108 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:32.364439 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:32.364801 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 16:48:32.365056 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 16:48:32.366146 22609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:32.464510 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 22382
I20260812 16:48:32.506459 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 22628
I20260812 16:48:32.526749 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:32.507471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.28.243.3:36151)
0812 16:48:32.507803 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:32.507811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:32.508592 (+ 781us) server_negotiation.cc:407] Connection header received
0812 16:48:32.508705 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:32.508710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:32.508819 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:32.509282 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:32.511354 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.512418 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.514000 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:32.514453 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:32.514571 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:32.518287 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:32.518308 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:32.518330 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:32.518379 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:32.522888 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:32.524001 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:32.524011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:32.524013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:32.524200 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:32.524751 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:32.524755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:32.524758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:32.525244 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:32.525789 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:32.526124 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 16:48:32.526414 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 16:48:32.527731 22844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:32.631497 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 22628
I20260812 16:48:32.665876 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 21964
I20260812 16:48:32.667340 22078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:32.789027 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 21964
I20260812 16:48:32.824075 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40897
--webserver_interface=127.28.243.62
--webserver_port=38469
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:33.236752 22935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:33.237098 22935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:33.237259 22935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:33.245867 22935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:33.246083 22935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:33.246142 22935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:33.246244 22935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:33.246300 22935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:33.258945 22935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:33.260987 22935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:33.263208 22935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:33.277380 22940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:33.277402 22941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:33.277534 22944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:33.278281 22935 server_base.cc:1034] running on GCE node
I20260812 16:48:33.279204 22935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:33.280912 22935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:33.282269 22935 hybrid_clock.cc:648] HybridClock initialized: now 1786553313282190 us; error 63 us; skew 500 ppm
I20260812 16:48:33.282702 22935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:33 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:33.287451 22935 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:33.289868 22935 webserver.cc:460] Webserver started at http://127.28.243.62:38469/ using document root <none> and password file <none>
I20260812 16:48:33.290731 22935 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:33.290863 22935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:33.297746 22935 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:48:33.304792 22954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:33.306807 22935 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:48:33.307070 22935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "3195de405bf74af98602ecf31e8556f1"
format_stamp: "Formatted at 2026-08-12 16:48:26 on dist-test-slave-1vmg"
server_key: "ff98f63a8d5c96f0f92caf953557a922"
server_key_iv: "6c28efbc62a189aa957e085c20ccd416"
server_key_version: "encryptionkey@0"
I20260812 16:48:33.308350 22935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:33.370754 22935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:33.377225 22935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:33.377657 22935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:33.400138 22935 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:40897
I20260812 16:48:33.400153 23053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:40897 every 8 connection(s)
I20260812 16:48:33.401832 22935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:33.411384 23054 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:48:33.412341 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 22935
I20260812 16:48:33.413187 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:44217
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:40897
--webserver_port=37643
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:33.415570 23054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap starting.
I20260812 16:48:33.440793 23054 log.cc:824] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:33.463627 23054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:48:33.464283 23054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1: Bootstrap complete.
I20260812 16:48:33.475965 23054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:33.476977 23054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3195de405bf74af98602ecf31e8556f1, State: Initialized, Role: FOLLOWER
I20260812 16:48:33.477732 23054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:33.477979 23054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:33.478107 23054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:33.478273 23054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 2 FOLLOWER]: Advancing to term 3
I20260812 16:48:33.484236 23054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:33.485175 23054 leader_election.cc:302] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3195de405bf74af98602ecf31e8556f1; no voters:
I20260812 16:48:33.485918 23054 leader_election.cc:288] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 16:48:33.486091 23060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 16:48:33.487303 23060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [term 3 LEADER]: Becoming Leader. State: Replica: 3195de405bf74af98602ecf31e8556f1, State: Running, Role: LEADER
I20260812 16:48:33.488054 23060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } }
I20260812 16:48:33.488538 23054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:33.491062 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3195de405bf74af98602ecf31e8556f1. Latest consensus state: current_term: 3 leader_uuid: "3195de405bf74af98602ecf31e8556f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } } }
I20260812 16:48:33.491534 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:33.491087 23061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3195de405bf74af98602ecf31e8556f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195de405bf74af98602ecf31e8556f1" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40897 } } }
I20260812 16:48:33.492303 23061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195de405bf74af98602ecf31e8556f1 [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:33.501895 23069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:33.514801 23069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:33.517066 23069 catalog_manager.cc:1259] Loaded cluster ID: 8aeeade267524bfd9e0e98dd00fce30b
I20260812 16:48:33.517285 23069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 16:48:33.521446 23069 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f120fa129ad google::LogMessage::Fail() at ??:0
@ 0x7f120fa1588a google::LogMessage::SendToLog() at ??:0
@ 0x7f120fa124de google::LogMessage::Flush() at ??:0
@ 0x7f120fa13a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1225449512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1225446976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f12254ce610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f12254ce439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f121265e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f121266191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1212661799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f121263ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12159166db start_thread at ??:0
@ 0x7f120e8ac61f clone at ??:0
W20260812 16:48:33.898553 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:33.898905 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:33.899063 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:33.907199 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:33.907415 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:33.907616 23058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:48:33.919214 23058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:44217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=37643
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:33.920878 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:33.922996 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:33.940083 23108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:33.939787 23124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:33.940100 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:33.945441 23058 server_base.cc:1034] running on GCE node
I20260812 16:48:33.946477 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:33.947989 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:33.949312 23058 hybrid_clock.cc:648] HybridClock initialized: now 1786553313949208 us; error 89 us; skew 500 ppm
I20260812 16:48:33.949772 23058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:33 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:33.954401 23058 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:48:33.956954 23058 webserver.cc:460] Webserver started at http://127.28.243.1:37643/ using document root <none> and password file <none>
I20260812 16:48:33.958048 23058 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:33.958205 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:33.966776 23058 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 16:48:33.971282 23135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:33.973196 23058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:48:33.973399 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "c3767dcce4084e5d92d473b9cdee32a6"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "3fadf483badcfe5e02d86f3e94b577e7"
server_key_iv: "acebc2005acc226c8c218d15598cc4e1"
server_key_version: "encryptionkey@0"
I20260812 16:48:33.974570 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:33.996817 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:34.002103 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:34.002414 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:34.004053 23058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:34.006139 23058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:34.006256 23058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:34.006371 23058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:34.006448 23058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:34.046965 23058 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:44217
I20260812 16:48:34.047250 23317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:44217 every 8 connection(s)
I20260812 16:48:34.049736 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:48:34.053803 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 23058
I20260812 16:48:34.054498 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:44735
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:40897
--webserver_port=40197
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:34.055720 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:34.053333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:57901)
0812 16:48:34.054277 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:34.054572 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":397,"threads_started":1}
W20260812 16:48:34.056643 23318 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:40897 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.62:40897: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
I20260812 16:48:34.057751 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:34.057326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:51391)
0812 16:48:34.057537 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:34.057618 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 16:48:34.058703 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:34.058336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:52089)
0812 16:48:34.058484 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:34.058566 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:48:34.058988 23318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:48:34.482389 23324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:34.482745 23324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:34.482858 23324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:34.493026 23324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:34.493192 23324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:34.493362 23324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:48:34.506083 23324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:44735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=40197
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:34.507673 23324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:34.509181 23324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:34.524964 23330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:34.525470 23329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:34.525517 23333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:34.526356 23324 server_base.cc:1034] running on GCE node
I20260812 16:48:34.527769 23324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:34.529290 23324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:34.530633 23324 hybrid_clock.cc:648] HybridClock initialized: now 1786553314530542 us; error 78 us; skew 500 ppm
I20260812 16:48:34.531185 23324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:34 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:34.534852 23324 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:48:34.536952 23324 webserver.cc:460] Webserver started at http://127.28.243.2:40197/ using document root <none> and password file <none>
I20260812 16:48:34.537806 23324 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:34.537937 23324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:34.545282 23324 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:48:34.549718 23342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:34.552337 23324 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 16:48:34.552631 23324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "2c102cfc2f5a4f1aa83ef11c6e761ce4"
format_stamp: "Formatted at 2026-08-12 16:48:27 on dist-test-slave-1vmg"
server_key: "638b4337670844982fdf71b1ae25dfad"
server_key_iv: "1ff34971b9ab2b3bab2eec2611955c0c"
server_key_version: "encryptionkey@0"
I20260812 16:48:34.554031 23324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:34.569013 23324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:34.575276 23324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:34.575620 23324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:34.576848 23324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:34.578519 23324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:34.578645 23324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:34.578778 23324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:34.578857 23324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:34.617640 23324 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:44735
I20260812 16:48:34.617710 23539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:44735 every 8 connection(s)
I20260812 16:48:34.619410 23324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:48:34.622337 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 23324
I20260812 16:48:34.623469 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:36151
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:40897
--webserver_port=39645
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:34.626744 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:34.623925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:44205)
0812 16:48:34.625345 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:34.625697 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":494,"threads_started":1}
W20260812 16:48:34.627779 23541 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:40897 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.62:40897: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
I20260812 16:48:34.628898 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:34.628474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:57153)
0812 16:48:34.628679 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:34.628772 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 16:48:34.630260 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:34.629788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:54311)
0812 16:48:34.630019 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:34.630113 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260812 16:48:34.630585 23541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:48:35.038035 23546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:35.038393 23546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:35.038514 23546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:35.046475 23546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:35.046633 23546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:35.046802 23546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:48:35.059686 23546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:36151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=39645
--tserver_master_addrs=127.28.243.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:35.061182 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:35.059747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:40669)
0812 16:48:35.060492 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:35.060617 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":453,"threads_started":1}
I20260812 16:48:35.061225 23546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:35.062760 23546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:35.077476 23561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:35.077418 23558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:35.077452 23559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:35.078408 23546 server_base.cc:1034] running on GCE node
I20260812 16:48:35.079114 23546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:35.080258 23546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:35.081508 23546 hybrid_clock.cc:648] HybridClock initialized: now 1786553315081442 us; error 53 us; skew 500 ppm
I20260812 16:48:35.082155 23546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:35 dist-test-slave-1vmg krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:35.086973 23546 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:48:35.089489 23546 webserver.cc:460] Webserver started at http://127.28.243.3:39645/ using document root <none> and password file <none>
I20260812 16:48:35.090382 23546 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:35.090538 23546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:35.097529 23546 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 16:48:35.101399 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:35.103973 23546 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 16:48:35.104285 23546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "49e45c009a3b4a23a48a7a4e24c1f03c"
format_stamp: "Formatted at 2026-08-12 16:48:28 on dist-test-slave-1vmg"
server_key: "84e072b8e7df2429f56ca2339a435a46"
server_key_iv: "e2237be204fc45f1a77bfa525ed4c463"
server_key_version: "encryptionkey@0"
I20260812 16:48:35.106204 23546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:35.129778 23546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:35.135748 23546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:35.136054 23546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:35.137296 23546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:35.139051 23546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:35.139168 23546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:35.139294 23546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:35.139366 23546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:35.181007 23546 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:36151
I20260812 16:48:35.181037 23778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:36151 every 8 connection(s)
I20260812 16:48:35.183259 23546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:48:35.185660 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 23546
I20260812 16:48:35.190570 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:35.187872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:50543)
0812 16:48:35.189127 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:35.189466 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":466,"threads_started":1}
W20260812 16:48:35.191674 23780 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:40897 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.62:40897: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
I20260812 16:48:35.192768 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:35.192359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:56455)
0812 16:48:35.192546 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:35.192630 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 16:48:35.193842 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:35.193442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:57431)
0812 16:48:35.193612 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:35.193708 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 16:48:35.194280 23780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 16:48:35.632685 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:35.631394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:54073)
0812 16:48:35.632154 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:35.632335 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260812 16:48:36.064137 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:36.063021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:44665)
0812 16:48:36.063778 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:36.063909 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":427,"threads_started":1}
I20260812 16:48:36.196199 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:36.195222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:39787)
0812 16:48:36.195750 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:36.195923 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":244,"threads_started":1}
I20260812 16:48:36.634855 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:36.633902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:59993)
0812 16:48:36.634547 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:36.634657 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":313,"threads_started":1}
I20260812 16:48:37.066738 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:37.065591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:58043)
0812 16:48:37.066280 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:37.066414 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":348,"threads_started":1}
I20260812 16:48:37.198153 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:37.197308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:55739)
0812 16:48:37.197867 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:37.197967 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":272,"threads_started":1}
I20260812 16:48:37.636839 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:37.635954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:33891)
0812 16:48:37.636513 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:37.636612 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":297,"threads_started":1}
I20260812 16:48:38.068912 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:38.068010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:34883)
0812 16:48:38.068612 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:38.068724 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":275,"threads_started":1}
I20260812 16:48:38.200225 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:38.199276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:50135)
0812 16:48:38.199821 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:38.199920 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
I20260812 16:48:38.638831 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:38.638014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:41037)
0812 16:48:38.638545 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:38.638635 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
I20260812 16:48:39.070880 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:39.070034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:55655)
0812 16:48:39.070578 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:39.070677 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":276,"threads_started":1}
I20260812 16:48:39.202540 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:39.201396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:43137)
0812 16:48:39.202090 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:39.202255 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
I20260812 16:48:39.640844 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:39.640013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:33357)
0812 16:48:39.640573 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:39.640661 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":328,"threads_started":1}
I20260812 16:48:40.073068 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:40.072229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:48783)
0812 16:48:40.072782 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:40.072893 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":283,"threads_started":1}
I20260812 16:48:40.205174 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:40.203992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:47887)
0812 16:48:40.204724 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:40.204840 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":435,"threads_started":1}
I20260812 16:48:40.644485 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:40.643274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:57689)
0812 16:48:40.644061 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:40.644217 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":354,"threads_started":1}
I20260812 16:48:41.075270 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:41.074330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:47779)
0812 16:48:41.074889 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:41.075034 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
I20260812 16:48:41.207513 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:41.206489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:44137)
0812 16:48:41.207215 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:41.207329 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"spinlock_wait_cycles":31616,"thread_start_us":508,"threads_started":1}
I20260812 16:48:41.647099 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:41.645845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:58877)
0812 16:48:41.646683 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:41.646790 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":441,"threads_started":1}
I20260812 16:48:42.077339 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:42.076500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:48259)
0812 16:48:42.077058 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:42.077158 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":298,"threads_started":1}
I20260812 16:48:42.210290 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:42.208586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:49361)
0812 16:48:42.209914 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:42.210032 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":454,"threads_started":1}
I20260812 16:48:42.648995 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:42.648228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:42153)
0812 16:48:42.648735 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:42.648821 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":261,"threads_started":1}
I20260812 16:48:43.079536 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:43.078542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:44909)
0812 16:48:43.079225 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:43.079332 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":287,"threads_started":1}
I20260812 16:48:43.212518 23822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:43.211685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:45445)
0812 16:48:43.212254 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:43.212349 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260812 16:48:43.651567 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:43.650478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:34593)
0812 16:48:43.651249 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:43.651364 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":404,"threads_started":1}
I20260812 16:48:44.081665 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:44.080761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:59501)
0812 16:48:44.081375 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:44.081473 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":539,"threads_started":1}
I20260812 16:48:44.214421 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:44.213490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:60833)
0812 16:48:44.214119 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:44.214226 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260812 16:48:44.653658 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:44.652853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:40021)
0812 16:48:44.653385 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:44.653488 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":297,"threads_started":1}
I20260812 16:48:45.084302 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:45.083083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:58807)
0812 16:48:45.083937 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:45.084063 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":504,"threads_started":1}
I20260812 16:48:45.216730 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:45.215533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:42849)
0812 16:48:45.216279 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:45.216518 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":392,"threads_started":1}
I20260812 16:48:45.656515 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:45.655405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:38629)
0812 16:48:45.656133 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:45.656268 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":332,"threads_started":1}
I20260812 16:48:46.087193 23837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:46.086028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:47753)
0812 16:48:46.086708 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:46.086878 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":350,"threads_started":1}
I20260812 16:48:46.218813 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:46.217929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:46695)
0812 16:48:46.218503 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:46.218615 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":281,"threads_started":1}
I20260812 16:48:46.658524 23839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:46.657711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:41257)
0812 16:48:46.658213 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:46.658310 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260812 16:48:47.089282 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:47.088449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:35159)
0812 16:48:47.088963 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:47.089065 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":251,"threads_started":1}
I20260812 16:48:47.221272 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:47.220110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:55255)
0812 16:48:47.220891 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:47.221045 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":264,"threads_started":1}
I20260812 16:48:47.660691 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:47.659702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:46201)
0812 16:48:47.660333 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:47.660462 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":307,"threads_started":1}
I20260812 16:48:48.091437 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:48.090482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:46313)
0812 16:48:48.091119 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:48.091231 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":324,"threads_started":1}
I20260812 16:48:48.224277 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:48.222826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:44319)
0812 16:48:48.223707 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:48.224011 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":501,"threads_started":1}
I20260812 16:48:48.662647 23851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:48.661779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:59435)
0812 16:48:48.662333 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:48.662441 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":267,"threads_started":1}
I20260812 16:48:49.093691 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:49.092851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:50457)
0812 16:48:49.093396 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:49.093485 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":284,"threads_started":1}
I20260812 16:48:49.226694 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:49.225647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:34943)
0812 16:48:49.226301 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:49.226428 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":356,"threads_started":1}
I20260812 16:48:49.664923 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:49.664149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.2:34153)
0812 16:48:49.664636 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:49.664744 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":246,"threads_started":1}
I20260812 16:48:50.096293 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:50.095273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.1:51191)
0812 16:48:50.095979 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:50.096104 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":361,"threads_started":1}
I20260812 16:48:50.189962 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 23058
I20260812 16:48:50.216248 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:50.191747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.28.243.1:44217)
0812 16:48:50.192541 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:50.192556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:50.192613 (+ 57us) server_negotiation.cc:407] Connection header received
0812 16:48:50.194272 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:50.194282 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:50.194746 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:50.195111 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:50.196665 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:50.198026 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:50.199440 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:50.200221 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:50.200384 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:50.208709 (+ 8325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:50.208720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:50.208740 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:50.208792 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:50.211905 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:50.212898 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:50.212908 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:50.212916 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:50.213154 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:50.213723 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:50.213730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:50.213733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:50.214618 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:50.214965 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:50.215458 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 16:48:50.215887 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":449,"threads_started":1}
I20260812 16:48:50.217861 23313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:50.230114 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:50.228445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40897 (local address 127.28.243.3:54993)
0812 16:48:50.229523 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:50.229736 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:40897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":753,"threads_started":1}
I20260812 16:48:50.317790 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 23058
I20260812 16:48:50.353266 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 23324
I20260812 16:48:50.376611 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:50.354517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.28.243.2:44735)
0812 16:48:50.355460 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:50.355475 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:50.355536 (+ 61us) server_negotiation.cc:407] Connection header received
0812 16:48:50.357381 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:50.357393 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:50.357896 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:50.358247 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:50.360261 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:50.361776 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:50.363731 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:50.364539 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:50.364720 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:50.368520 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:50.368534 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:50.368561 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:50.368630 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:50.372363 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:50.373599 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:50.373608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:50.373616 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:50.373858 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:50.374343 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:50.374347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:50.374349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:50.375026 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:50.375301 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:50.375831 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 16:48:50.376308 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":580,"threads_started":1}
I20260812 16:48:50.378109 23533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:50.475342 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 23324
I20260812 16:48:50.503603 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 23546
I20260812 16:48:50.524979 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:50.505096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.28.243.3:36151)
0812 16:48:50.505961 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:50.505977 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:50.506037 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:48:50.507883 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:50.507895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:50.508379 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:50.508731 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:50.510393 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:50.512035 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:50.513467 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:50.514252 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:50.514461 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:50.517498 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:50.517507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:50.517528 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:50.517579 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:50.521087 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:50.522009 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:50.522018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:50.522026 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:50.522249 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:50.522675 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:50.522679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:50.522681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:50.523353 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:50.523682 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:50.524144 (+ 462us) server_negotiation.cc:299] Negotiation successful
0812 16:48:50.524624 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":405,"threads_started":1}
I20260812 16:48:50.526569 23772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:50.617419 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 23546
2026-08-12T16:48:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26385 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:48:50 dist-test-slave-1vmg krb5kdc[23876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:48:50 dist-test-slave-1vmg krb5kdc[23876](info): set up 2 sockets
Aug 12 16:48:50 dist-test-slave-1vmg krb5kdc[23876](info): commencing operation
krb5kdc: starting...
W20260812 16:48:52.028589 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.358s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:48:52 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:48:52.384740 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38365
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:38365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:52.775329 23897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:52.775686 23897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:52.775835 23897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:52.784240 23897 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 16:48:52.784387 23897 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 16:48:52.784456 23897 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 16:48:52.784525 23897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:48:52.784600 23897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:52.784839 23897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:52.784936 23897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:48:52.785025 23897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:48:52.796662 23897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:38365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:52.798611 23897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:52.800575 23897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:52.815932 23911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:52.815804 23917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:52.815897 23912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:52.816753 23897 server_base.cc:1034] running on GCE node
I20260812 16:48:52.817669 23897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:52.819265 23897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:52.820608 23897 hybrid_clock.cc:648] HybridClock initialized: now 1786553332820508 us; error 59 us; skew 500 ppm
I20260812 16:48:52.821023 23897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:52 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:52.824853 23897 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:48:52.826546 23897 webserver.cc:460] Webserver started at http://127.28.243.62:37731/ using document root <none> and password file <none>
I20260812 16:48:52.827394 23897 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:52.827514 23897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:52.827888 23897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:52.831185 23897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb"
format_stamp: "Formatted at 2026-08-12 16:48:52 on dist-test-slave-1vmg"
server_key: "eb0936d87d1e1ea0da8ccde2f0d3c779"
server_key_iv: "0146cff4ba2086e1a7ab9d33eba125e3"
server_key_version: "encryptionkey@0"
I20260812 16:48:52.832019 23897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb"
format_stamp: "Formatted at 2026-08-12 16:48:52 on dist-test-slave-1vmg"
server_key: "eb0936d87d1e1ea0da8ccde2f0d3c779"
server_key_iv: "0146cff4ba2086e1a7ab9d33eba125e3"
server_key_version: "encryptionkey@0"
I20260812 16:48:52.837982 23897 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:48:52.842749 23927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:52.844473 23897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:48:52.844698 23897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb"
format_stamp: "Formatted at 2026-08-12 16:48:52 on dist-test-slave-1vmg"
server_key: "eb0936d87d1e1ea0da8ccde2f0d3c779"
server_key_iv: "0146cff4ba2086e1a7ab9d33eba125e3"
server_key_version: "encryptionkey@0"
I20260812 16:48:52.844947 23897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:52.863026 23897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:52.868238 23897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:52.868575 23897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:52.887810 23897 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:38365
I20260812 16:48:52.887821 24012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:38365 every 8 connection(s)
I20260812 16:48:52.889418 23897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:48:52.893883 24013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:52.899610 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 23897
I20260812 16:48:52.899865 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:48:52.900559 29644 external_mini_cluster.cc:1428] Setting key c1231cf25734348af0a6e7c8daf9ed53
I20260812 16:48:52.906864 24013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap starting.
I20260812 16:48:52.910385 24013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:52.911598 24013 log.cc:824] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:52.915359 24013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: No bootstrap required, opened a new log
Aug 12 16:48:52 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:52.924723 24013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:48:52.925158 24013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:52.925290 24013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fcdfc82df4e4ba8b733f5d7fd5223eb, State: Initialized, Role: FOLLOWER
I20260812 16:48:52.925979 24013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:48:52.926266 24013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:48:52.926448 24013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:48:52.926689 24013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:52.927940 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:52.903070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.28.243.62:38365)
0812 16:48:52.904089 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:52.904106 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:52.904157 (+ 51us) server_negotiation.cc:407] Connection header received
0812 16:48:52.905962 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:52.905989 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:52.906437 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:52.906870 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:52.908784 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:52.910158 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:52.911846 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:52.913332 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:52.913567 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:52.919013 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:52.919028 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:52.919050 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:52.919112 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:52.922779 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:52.923968 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:52.923978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:52.923987 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:52.924215 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:52.924653 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:52.924657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:52.924659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:52.925341 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:52.925678 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:52.926195 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 16:48:52.926698 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":487,"threads_started":1}
I20260812 16:48:52.929422 24013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:48:52.929972 24013 leader_election.cc:302] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fcdfc82df4e4ba8b733f5d7fd5223eb; no voters:
I20260812 16:48:52.930656 24013 leader_election.cc:288] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:48:52.930794 24022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:52.931610 24022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 LEADER]: Becoming Leader. State: Replica: 7fcdfc82df4e4ba8b733f5d7fd5223eb, State: Running, Role: LEADER
I20260812 16:48:52.932335 24022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:48:52.933308 24013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:48:52.935925 24026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } } }
I20260812 16:48:52.936452 24026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:52.935599 24027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fcdfc82df4e4ba8b733f5d7fd5223eb. Latest consensus state: current_term: 1 leader_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } } }
I20260812 16:48:52.936864 24027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: This master's current role is: LEADER
I20260812 16:48:52.941072 24041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:48:52.945868 24041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:48:52.955343 24041 catalog_manager.cc:1347] Generated new cluster ID: 8c8c2f5a180948078a4661dccc53a6dc
I20260812 16:48:52.955489 24041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:48:52.984706 24041 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:48:52.986778 24041 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:48:52.995119 24041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Generated new TSK 0
I20260812 16:48:52.996192 24041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:48:53.100328 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:38365
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:53.544569 24062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:53.544899 24062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:53.545044 24062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:53.553153 24062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:53.553305 24062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:53.553457 24062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:48:53.564529 24062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:53.566041 24062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:53.567526 24062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:53.585757 24068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:53.585747 24067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:53.585744 24072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:53.589489 24062 server_base.cc:1034] running on GCE node
I20260812 16:48:53.590343 24062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:53.591886 24062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:53.593196 24062 hybrid_clock.cc:648] HybridClock initialized: now 1786553333593148 us; error 46 us; skew 500 ppm
I20260812 16:48:53.593636 24062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:53 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:53.597878 24062 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:48:53.600294 24062 webserver.cc:460] Webserver started at http://127.28.243.1:44805/ using document root <none> and password file <none>
I20260812 16:48:53.601353 24062 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:53.601493 24062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:53.601907 24062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:53.605962 24062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4"
format_stamp: "Formatted at 2026-08-12 16:48:53 on dist-test-slave-1vmg"
server_key: "f88327d7a9f0d1d4dfde559aa69c8de3"
server_key_iv: "61a2f433db6180715ed8dde530230d0b"
server_key_version: "encryptionkey@0"
I20260812 16:48:53.606894 24062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4"
format_stamp: "Formatted at 2026-08-12 16:48:53 on dist-test-slave-1vmg"
server_key: "f88327d7a9f0d1d4dfde559aa69c8de3"
server_key_iv: "61a2f433db6180715ed8dde530230d0b"
server_key_version: "encryptionkey@0"
I20260812 16:48:53.612841 24062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:48:53.617062 24082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:53.618817 24062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:48:53.619068 24062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4"
format_stamp: "Formatted at 2026-08-12 16:48:53 on dist-test-slave-1vmg"
server_key: "f88327d7a9f0d1d4dfde559aa69c8de3"
server_key_iv: "61a2f433db6180715ed8dde530230d0b"
server_key_version: "encryptionkey@0"
I20260812 16:48:53.619325 24062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:53.638899 24062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:53.644299 24062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:53.644630 24062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:53.645735 24062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:53.647687 24062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:53.647785 24062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:53.647902 24062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:53.647974 24062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:53.686685 24062 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:35657
I20260812 16:48:53.686707 24271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:35657 every 8 connection(s)
I20260812 16:48:53.688119 24062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:48:53.691931 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 24062
I20260812 16:48:53.692155 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:48:53.692827 29644 external_mini_cluster.cc:1428] Setting key d2a90dfd83dafbfef5f47fb08cb6a7c9
Aug 12 16:48:53 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553333, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:53.714797 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:53.693836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:33761 (local address 127.28.243.62:38365)
0812 16:48:53.695428 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:53.695435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:53.695462 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:48:53.696705 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:53.696712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:53.696828 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:53.696934 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:53.698769 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:53.699619 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:53.701556 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:53.701998 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:53.702125 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:53.707743 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:53.707749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:53.707753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:53.707802 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:53.711095 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:53.712053 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:53.712059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:53.712062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:53.712264 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:53.712800 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:53.712804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:53.712807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:53.713196 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:53.713351 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:53.714161 (+ 810us) server_negotiation.cc:299] Negotiation successful
0812 16:48:53.714438 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":1296,"threads_started":1}
I20260812 16:48:53.715762 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:53.692539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:33761)
0812 16:48:53.693757 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:53.693827 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:53.696343 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:53.697257 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:53.697268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:53.697876 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:53.698531 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:53.698552 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:53.699818 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:53.699823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:53.701368 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:53.701385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:53.702269 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:53.702276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:53.702853 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:53.703981 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:53.704015 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:53.707426 (+ 3411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:53.711337 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:53.711348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:53.711365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:53.711899 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:53.712416 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:53.712421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:53.712423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:53.712677 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:53.713353 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:53.713361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:53.713764 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 16:48:53.714034 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 16:48:53.714492 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":422,"threads_started":1}
I20260812 16:48:53.717945 24275 heartbeater.cc:347] Connected to a master server at 127.28.243.62:38365
I20260812 16:48:53.718423 24275 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:53.719606 24275 heartbeater.cc:511] Master 127.28.243.62:38365 requested a full tablet report, sending...
I20260812 16:48:53.722256 23953 ts_manager.cc:194] Registered new tserver with Master: 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1:35657)
I20260812 16:48:53.723865 23953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:33761
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:48:53.771659 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:38365
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:48:54.182782 24287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:54.183117 24287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:54.183408 24287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:54.191009 24287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:54.191128 24287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:54.191290 24287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:48:54.201326 24287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:54.202648 24287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:54.204025 24287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:54.218176 24300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:54.218170 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:54.218184 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:54.219704 24287 server_base.cc:1034] running on GCE node
I20260812 16:48:54.220474 24287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:54.222204 24287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:54.223516 24287 hybrid_clock.cc:648] HybridClock initialized: now 1786553334223440 us; error 57 us; skew 500 ppm
I20260812 16:48:54.223891 24287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:54 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:54.228037 24287 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:48:54.230171 24287 webserver.cc:460] Webserver started at http://127.28.243.2:34073/ using document root <none> and password file <none>
I20260812 16:48:54.231009 24287 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:54.231133 24287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:54.231442 24287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:54.234715 24287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "d8df6f3d96ae45a59f26d124dddc396f"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "bd2c781ec463cafd69dfcea6f251e959"
server_key_iv: "573f815951845060de38a7db72441ee2"
server_key_version: "encryptionkey@0"
I20260812 16:48:54.235594 24287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "d8df6f3d96ae45a59f26d124dddc396f"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "bd2c781ec463cafd69dfcea6f251e959"
server_key_iv: "573f815951845060de38a7db72441ee2"
server_key_version: "encryptionkey@0"
I20260812 16:48:54.241775 24287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:48:54.251314 24315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:54.253206 24287 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:48:54.253437 24287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "d8df6f3d96ae45a59f26d124dddc396f"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "bd2c781ec463cafd69dfcea6f251e959"
server_key_iv: "573f815951845060de38a7db72441ee2"
server_key_version: "encryptionkey@0"
I20260812 16:48:54.253660 24287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:54.271746 24287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:54.276455 24287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:54.276758 24287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:54.277817 24287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:54.279383 24287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:54.279472 24287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:54.279603 24287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:54.279675 24287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:54.318768 24287 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:41195
I20260812 16:48:54.319008 24508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:41195 every 8 connection(s)
I20260812 16:48:54.320315 24287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:48:54.329607 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 24287
I20260812 16:48:54.329829 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:48:54.330390 29644 external_mini_cluster.cc:1428] Setting key 97065234ee49e0d743f5e48cd87bc373
Aug 12 16:48:54 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:54.346047 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:54.323975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:36533 (local address 127.28.243.62:38365)
0812 16:48:54.324465 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:54.324469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:54.326175 (+ 1706us) server_negotiation.cc:407] Connection header received
0812 16:48:54.328357 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:54.328362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:54.328460 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:54.328561 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:54.330762 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.331693 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.334041 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.334591 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.334706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:54.339340 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:54.339346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:54.339350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:54.339389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:54.342211 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:54.343334 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:54.343340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:54.343342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:54.343493 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:54.344084 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:54.344088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:54.344091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:54.344434 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:54.344580 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:54.345474 (+ 894us) server_negotiation.cc:299] Negotiation successful
0812 16:48:54.345703 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
I20260812 16:48:54.347355 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:54.324760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:36533)
0812 16:48:54.325968 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:54.326021 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:54.328022 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:54.328842 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:54.328851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:54.329621 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:54.330498 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:54.330528 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.331869 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.331874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:54.333823 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:54.333846 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.334865 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.334874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:54.335230 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:54.336059 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:54.336091 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:54.339037 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:54.342410 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:54.342420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:54.342443 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:54.343109 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:54.343612 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:54.343617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:54.343631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:54.343924 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:54.344575 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:54.344584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:54.345069 (+ 485us) client_negotiation.cc:769] Sending connection context
0812 16:48:54.345381 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 16:48:54.345960 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":376,"threads_started":1}
I20260812 16:48:54.348887 24511 heartbeater.cc:347] Connected to a master server at 127.28.243.62:38365
I20260812 16:48:54.349232 24511 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:54.349989 24511 heartbeater.cc:511] Master 127.28.243.62:38365 requested a full tablet report, sending...
I20260812 16:48:54.351544 23953 ts_manager.cc:194] Registered new tserver with Master: d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195)
I20260812 16:48:54.352388 23953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:36533
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:48:54.402511 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:38365
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:48:54.727166 24275 heartbeater.cc:503] Master 127.28.243.62:38365 was elected leader, sending a full tablet report...
W20260812 16:48:54.785109 24522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:48:54.785495 24522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:48:54.785629 24522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:48:54.793581 24522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:48:54.793722 24522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:48:54.793874 24522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:48:54.804698 24522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:48:54.806169 24522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:48:54.807662 24522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:48:54.821892 24535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:54.822032 24543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:48:54.822032 24537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:48:54.823370 24522 server_base.cc:1034] running on GCE node
I20260812 16:48:54.824524 24522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:48:54.825726 24522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:48:54.827072 24522 hybrid_clock.cc:648] HybridClock initialized: now 1786553334827012 us; error 44 us; skew 500 ppm
I20260812 16:48:54.827426 24522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:48:54 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:48:54.830757 24522 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:48:54.832428 24522 webserver.cc:460] Webserver started at http://127.28.243.3:33717/ using document root <none> and password file <none>
I20260812 16:48:54.833189 24522 fs_manager.cc:359] Metadata directory not provided
I20260812 16:48:54.833323 24522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:48:54.833675 24522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:48:54.837373 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "56c351cc8fc2413691f1f3bb42dcf454"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "d5a195af1442eb7247e72e36d4f10196"
server_key_iv: "bffb9c08d6ddb7bd2ef6d5a20f74b47c"
server_key_version: "encryptionkey@0"
I20260812 16:48:54.838282 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "56c351cc8fc2413691f1f3bb42dcf454"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "d5a195af1442eb7247e72e36d4f10196"
server_key_iv: "bffb9c08d6ddb7bd2ef6d5a20f74b47c"
server_key_version: "encryptionkey@0"
I20260812 16:48:54.844245 24522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:48:54.847884 24555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:54.849507 24522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:48:54.849701 24522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "56c351cc8fc2413691f1f3bb42dcf454"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "d5a195af1442eb7247e72e36d4f10196"
server_key_iv: "bffb9c08d6ddb7bd2ef6d5a20f74b47c"
server_key_version: "encryptionkey@0"
I20260812 16:48:54.849901 24522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:48:54.876624 24522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:48:54.881778 24522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:48:54.882081 24522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:48:54.883245 24522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:48:54.884793 24522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:48:54.884876 24522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:54.884963 24522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:48:54.885008 24522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:48:54.925132 24522 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:40495
I20260812 16:48:54.925154 24768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:40495 every 8 connection(s)
I20260812 16:48:54.926550 24522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:48:54.928016 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 24522
I20260812 16:48:54.928198 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:48:54.928702 29644 external_mini_cluster.cc:1428] Setting key ff8bbf853e68c1586dcd041cfedb2bbc
Aug 12 16:48:54 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:48:54.950819 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:54.929604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:59667 (local address 127.28.243.62:38365)
0812 16:48:54.930102 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:54.930108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:54.931280 (+ 1172us) server_negotiation.cc:407] Connection header received
0812 16:48:54.933435 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:54.933439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:54.933515 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:54.933625 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:54.935663 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.936447 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.938242 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.938764 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.938875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:54.943431 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:54.943437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:54.943441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:54.943485 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:54.946508 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:54.947515 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:54.947520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:54.947522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:54.947662 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:54.948518 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:54.948522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:54.948525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:54.948995 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:54.949149 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:54.950268 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0812 16:48:54.950497 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260812 16:48:54.951865 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:54.930098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:59667)
0812 16:48:54.931089 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:54.931148 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:54.933099 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:54.934143 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:54.934152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:54.934745 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:48:54.935476 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:54.935496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.936639 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.936643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:54.938052 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:54.938068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.939043 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.939050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:54.939279 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:54.939988 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:48:54.940013 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:48:54.943093 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:48:54.946694 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:54.946703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:54.946721 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:54.947343 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:54.947909 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:48:54.947915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:48:54.947919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:48:54.948303 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:48:54.949196 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:48:54.949205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:48:54.949654 (+ 449us) client_negotiation.cc:769] Sending connection context
0812 16:48:54.950131 (+ 477us) client_negotiation.cc:240] Negotiation successful
0812 16:48:54.950642 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":393,"threads_started":1}
I20260812 16:48:54.953624 24770 heartbeater.cc:347] Connected to a master server at 127.28.243.62:38365
I20260812 16:48:54.954094 24770 heartbeater.cc:464] Registering TS with master...
I20260812 16:48:54.955013 24770 heartbeater.cc:511] Master 127.28.243.62:38365 requested a full tablet report, sending...
I20260812 16:48:54.957852 23953 ts_manager.cc:194] Registered new tserver with Master: 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3:40495)
I20260812 16:48:54.959039 23953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:59667
I20260812 16:48:54.961112 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:48:54.982057 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:54.966362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.28.243.62:38365)
0812 16:48:54.966595 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:54.966603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:54.967038 (+ 435us) server_negotiation.cc:407] Connection header received
0812 16:48:54.967206 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:54.967210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:54.967300 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:54.967392 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:54.969146 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.969907 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.971503 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:54.971941 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:54.972056 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:54.974900 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:54.975021 (+ 121us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:54.975038 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:54.975098 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:54.977830 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:54.979077 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:54.979083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:54.979085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:54.979242 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:54.979797 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:54.979801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:54.979803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:54.980193 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:54.980402 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:54.981385 (+ 983us) server_negotiation.cc:299] Negotiation successful
0812 16:48:54.981750 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:48:54.987088 23953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:48:54.995025 23953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:48:55.035145 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.010783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.28.243.3:40495)
0812 16:48:55.024509 (+ 13726us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.024518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.025013 (+ 495us) server_negotiation.cc:407] Connection header received
0812 16:48:55.025250 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.025259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.025581 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.025808 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:55.026567 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.028251 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.031914 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.033646 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.033813 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.034080 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.034520 (+ 440us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.034721 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13462,"thread_start_us":13286,"threads_started":1}
I20260812 16:48:55.035197 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.010366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:41195 (local address 127.0.0.1:37400)
0812 16:48:55.024508 (+ 14142us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:55.024629 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:55.024851 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:55.026474 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:55.026479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:55.026508 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:55.027026 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.027045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.028791 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.028796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.031599 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.031624 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.034211 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.034216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.034365 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.034389 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:48:55.034476 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:48:55.034723 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13037,"mutex_wait_us":85,"thread_start_us":14685,"threads_started":1}
I20260812 16:48:55.036204 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.010363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.28.243.2:41195)
0812 16:48:55.025666 (+ 15303us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.025673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.025697 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:48:55.025798 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.025805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.026023 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.026240 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:55.027264 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.028610 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.031852 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.033646 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.033813 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.034044 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.034713 (+ 669us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.035875 (+ 1162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15150,"thread_start_us":15006,"threads_started":1}
I20260812 16:48:55.036213 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.025824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.28.243.1:35657)
0812 16:48:55.026529 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.026536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.027412 (+ 876us) server_negotiation.cc:407] Connection header received
0812 16:48:55.027601 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.027609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.027869 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.028069 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:55.028723 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.030308 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.033248 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.034657 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.034803 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.035080 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.035668 (+ 588us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.035852 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":390,"threads_started":1}
I20260812 16:48:55.035941 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.010384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:40495 (local address 127.0.0.1:44232)
0812 16:48:55.024861 (+ 14477us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:55.024905 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:55.025065 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:55.025836 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:55.025844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:55.025878 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:55.026340 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.026357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.028481 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.028487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.031534 (+ 3047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.031585 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.033899 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.033907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.034213 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.034278 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 16:48:55.034439 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 16:48:55.035519 (+ 1080us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14306,"mutex_wait_us":925,"thread_start_us":12878,"threads_started":1}
I20260812 16:48:55.036814 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.026556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35657 (local address 127.0.0.1:50384)
0812 16:48:55.027272 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:55.027291 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:55.027432 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:55.028087 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:55.028092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:55.028120 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:55.028485 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.028497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.030479 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.030485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.032988 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.033067 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.035256 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.035264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.035442 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.035472 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 16:48:55.036350 (+ 878us) client_negotiation.cc:240] Negotiation successful
0812 16:48:55.036509 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":404,"threads_started":1}
I20260812 16:48:55.040556 24168 tablet_service.cc:1467] Processing CreateTablet for tablet 386c626a6cc94c8a90fff97eeacf4768 (DEFAULT_TABLE table=test-table [id=7b878398c8fc47e3bfb6f5b449170569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:55.041097 24638 tablet_service.cc:1467] Processing CreateTablet for tablet 386c626a6cc94c8a90fff97eeacf4768 (DEFAULT_TABLE table=test-table [id=7b878398c8fc47e3bfb6f5b449170569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:55.041097 24402 tablet_service.cc:1467] Processing CreateTablet for tablet 386c626a6cc94c8a90fff97eeacf4768 (DEFAULT_TABLE table=test-table [id=7b878398c8fc47e3bfb6f5b449170569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:48:55.042636 24168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c626a6cc94c8a90fff97eeacf4768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:55.042693 24638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c626a6cc94c8a90fff97eeacf4768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:55.042711 24402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c626a6cc94c8a90fff97eeacf4768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:48:55.056108 24801 tablet_bootstrap.cc:492] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Bootstrap starting.
I20260812 16:48:55.060149 24801 tablet_bootstrap.cc:654] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:55.061990 24801 log.cc:824] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:55.064114 24803 tablet_bootstrap.cc:492] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Bootstrap starting.
I20260812 16:48:55.066776 24802 tablet_bootstrap.cc:492] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Bootstrap starting.
I20260812 16:48:55.068589 24803 tablet_bootstrap.cc:654] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:55.068792 24801 tablet_bootstrap.cc:492] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: No bootstrap required, opened a new log
I20260812 16:48:55.069090 24801 ts_tablet_manager.cc:1397] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260812 16:48:55.070075 24803 log.cc:824] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:55.070748 24802 tablet_bootstrap.cc:654] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Neither blocks nor log segments found. Creating new log.
I20260812 16:48:55.072296 24802 log.cc:824] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:48:55.073992 24803 tablet_bootstrap.cc:492] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: No bootstrap required, opened a new log
I20260812 16:48:55.074337 24803 ts_tablet_manager.cc:1397] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 16:48:55.075933 24802 tablet_bootstrap.cc:492] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: No bootstrap required, opened a new log
I20260812 16:48:55.076274 24802 ts_tablet_manager.cc:1397] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 16:48:55.078469 24801 raft_consensus.cc:348] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.078905 24801 raft_consensus.cc:374] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:55.079030 24801 raft_consensus.cc:729] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46986aa1aa3d4e40a9279dc6d5a9ecf4, State: Initialized, Role: FOLLOWER
I20260812 16:48:55.079730 24801 consensus_queue.cc:260] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.084661 24801 ts_tablet_manager.cc:1428] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Time spent starting tablet: real 0.015s user 0.002s sys 0.014s
I20260812 16:48:55.086159 24803 raft_consensus.cc:348] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.086719 24803 raft_consensus.cc:374] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:55.086836 24803 raft_consensus.cc:729] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c351cc8fc2413691f1f3bb42dcf454, State: Initialized, Role: FOLLOWER
I20260812 16:48:55.087383 24802 raft_consensus.cc:348] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.087587 24803 consensus_queue.cc:260] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.087859 24802 raft_consensus.cc:374] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:48:55.087977 24802 raft_consensus.cc:729] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8df6f3d96ae45a59f26d124dddc396f, State: Initialized, Role: FOLLOWER
I20260812 16:48:55.088733 24802 consensus_queue.cc:260] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.089901 24770 heartbeater.cc:503] Master 127.28.243.62:38365 was elected leader, sending a full tablet report...
I20260812 16:48:55.090734 24803 ts_tablet_manager.cc:1428] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260812 16:48:55.091779 24511 heartbeater.cc:503] Master 127.28.243.62:38365 was elected leader, sending a full tablet report...
I20260812 16:48:55.092481 24802 ts_tablet_manager.cc:1428] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260812 16:48:55.178613 24771 tablet.cc:2369] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:48:55.191457 24276 tablet.cc:2369] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:48:55.323069 24512 tablet.cc:2369] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:48:55.370569 24807 raft_consensus.cc:482] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:48:55.371279 24807 raft_consensus.cc:504] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.373248 24807 leader_election.cc:288] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195), 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3:40495)
I20260812 16:48:55.383285 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.373929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:48343 (local address 127.28.243.3:40495)
0812 16:48:55.374230 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.374237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.374694 (+ 457us) server_negotiation.cc:407] Connection header received
0812 16:48:55.374906 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.374933 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.375014 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.375089 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:55.375730 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.376832 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.379915 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.381222 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.381428 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.381512 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.382518 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.382742 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260812 16:48:55.383284 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.373786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:53653 (local address 127.28.243.2:41195)
0812 16:48:55.374016 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.374023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.374892 (+ 869us) server_negotiation.cc:407] Connection header received
0812 16:48:55.375100 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.375104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.375168 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.375246 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:48:55.375929 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.376952 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.379927 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.381253 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.381427 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.381512 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.382518 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.382742 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:48:55.384424 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.373747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:41195 (local address 127.28.243.1:53653)
0812 16:48:55.374774 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:55.374789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:55.374958 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:55.375278 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:55.375282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:55.375328 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:55.375754 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.375767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.377092 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.377100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.379686 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.379708 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.381978 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.382000 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.382217 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.382243 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 16:48:55.383120 (+ 877us) client_negotiation.cc:240] Negotiation successful
0812 16:48:55.383474 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":150,"thread_start_us":431,"threads_started":1}
I20260812 16:48:55.384491 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.373713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:40495 (local address 127.28.243.1:48343)
0812 16:48:55.374548 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0812 16:48:55.374575 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:48:55.374753 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:48:55.375138 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:48:55.375143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:48:55.375178 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:48:55.375576 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.375586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.377019 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.377025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.379686 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:48:55.379708 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.381744 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.381814 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:48:55.382145 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.382205 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 16:48:55.383596 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0812 16:48:55.383773 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":140,"thread_start_us":728,"threads_started":1}
I20260812 16:48:55.385953 24677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c626a6cc94c8a90fff97eeacf4768" candidate_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c351cc8fc2413691f1f3bb42dcf454" is_pre_election: true
I20260812 16:48:55.385952 24433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c626a6cc94c8a90fff97eeacf4768" candidate_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df6f3d96ae45a59f26d124dddc396f" is_pre_election: true
I20260812 16:48:55.386579 24677 raft_consensus.cc:2454] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46986aa1aa3d4e40a9279dc6d5a9ecf4 in term 0.
I20260812 16:48:55.386579 24433 raft_consensus.cc:2454] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46986aa1aa3d4e40a9279dc6d5a9ecf4 in term 0.
I20260812 16:48:55.387491 24085 leader_election.cc:302] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46986aa1aa3d4e40a9279dc6d5a9ecf4, 56c351cc8fc2413691f1f3bb42dcf454; no voters:
I20260812 16:48:55.387957 24807 raft_consensus.cc:2781] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:48:55.388134 24807 raft_consensus.cc:482] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:48:55.388243 24807 raft_consensus.cc:3037] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:55.392057 24807 raft_consensus.cc:504] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.392907 24807 leader_election.cc:288] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [CANDIDATE]: Term 1 election: Requested vote from peers d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195), 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3:40495)
I20260812 16:48:55.393548 24677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c626a6cc94c8a90fff97eeacf4768" candidate_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c351cc8fc2413691f1f3bb42dcf454"
I20260812 16:48:55.393553 24433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c626a6cc94c8a90fff97eeacf4768" candidate_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df6f3d96ae45a59f26d124dddc396f"
I20260812 16:48:55.393769 24677 raft_consensus.cc:3037] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:55.393785 24433 raft_consensus.cc:3037] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:48:55.396219 24433 raft_consensus.cc:2454] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46986aa1aa3d4e40a9279dc6d5a9ecf4 in term 1.
I20260812 16:48:55.396224 24677 raft_consensus.cc:2454] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46986aa1aa3d4e40a9279dc6d5a9ecf4 in term 1.
I20260812 16:48:55.396829 24085 leader_election.cc:302] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 46986aa1aa3d4e40a9279dc6d5a9ecf4, 56c351cc8fc2413691f1f3bb42dcf454, d8df6f3d96ae45a59f26d124dddc396f; no voters:
I20260812 16:48:55.397213 24807 raft_consensus.cc:2781] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:48:55.397725 24807 raft_consensus.cc:686] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 LEADER]: Becoming Leader. State: Replica: 46986aa1aa3d4e40a9279dc6d5a9ecf4, State: Running, Role: LEADER
I20260812 16:48:55.398234 24807 consensus_queue.cc:237] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } }
I20260812 16:48:55.403932 23953 catalog_manager.cc:5495] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1). New cstate: current_term: 1 leader_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } health_report { overall_health: HEALTHY } } }
I20260812 16:48:55.492143 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.482800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.28.243.1:35657)
0812 16:48:55.483102 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.483108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.484431 (+ 1323us) server_negotiation.cc:407] Connection header received
0812 16:48:55.484773 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.484779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.484930 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.485122 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:48:55.485781 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.486824 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.489068 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.489560 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.489691 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.490483 (+ 792us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:48:55.491277 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:48:55.491417 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.491694 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.491832 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 16:48:55.502310 24677 raft_consensus.cc:1261] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Refusing update from remote peer 46986aa1aa3d4e40a9279dc6d5a9ecf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:55.502310 24433 raft_consensus.cc:1261] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Refusing update from remote peer 46986aa1aa3d4e40a9279dc6d5a9ecf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:48:55.503453 24817 consensus_queue.cc:1035] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:55.503851 24807 consensus_queue.cc:1035] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:48:55.534647 23953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:55.535022 23953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:48:55.538968 23953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Sending DeleteTablet for 3 replicas of tablet 386c626a6cc94c8a90fff97eeacf4768
I20260812 16:48:55.540437 24638 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c626a6cc94c8a90fff97eeacf4768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:55 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:44232
I20260812 16:48:55.540823 24402 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c626a6cc94c8a90fff97eeacf4768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:55 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:37400
I20260812 16:48:55.540808 24168 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c626a6cc94c8a90fff97eeacf4768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:48:55 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:50384
I20260812 16:48:55.541750 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 24062
I20260812 16:48:55.554349 24844 tablet_replica.cc:330] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: stopping tablet replica
I20260812 16:48:55.555336 24841 tablet_replica.cc:330] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: stopping tablet replica
I20260812 16:48:55.555991 24841 raft_consensus.cc:2229] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:55.556632 24841 raft_consensus.cc:2258] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:55.557363 24842 tablet_replica.cc:330] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: stopping tablet replica
I20260812 16:48:55.557806 24842 raft_consensus.cc:2229] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:48:55.558172 24844 raft_consensus.cc:2229] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:48:55.558655 24842 raft_consensus.cc:2258] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:55.560025 24844 raft_consensus.cc:2258] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:48:55.561926 24841 ts_tablet_manager.cc:1905] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:55.563628 24844 ts_tablet_manager.cc:1905] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:55.571138 24841 ts_tablet_manager.cc:1918] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:55.571769 24841 log.cc:1196] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/386c626a6cc94c8a90fff97eeacf4768
I20260812 16:48:55.572153 24844 ts_tablet_manager.cc:1918] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:55.572530 24844 log.cc:1196] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/386c626a6cc94c8a90fff97eeacf4768
I20260812 16:48:55.572567 24842 ts_tablet_manager.cc:1905] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:48:55.572839 24841 ts_tablet_manager.cc:1939] T 386c626a6cc94c8a90fff97eeacf4768 P 56c351cc8fc2413691f1f3bb42dcf454: Deleting consensus metadata
I20260812 16:48:55.573364 24844 ts_tablet_manager.cc:1939] T 386c626a6cc94c8a90fff97eeacf4768 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Deleting consensus metadata
Aug 12 16:48:55 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:48:55.575558 23929 catalog_manager.cc:4841] TS 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3:40495): tablet 386c626a6cc94c8a90fff97eeacf4768 (table test-table [id=7b878398c8fc47e3bfb6f5b449170569]) successfully deleted
I20260812 16:48:55.575645 23933 catalog_manager.cc:4841] TS 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1:35657): tablet 386c626a6cc94c8a90fff97eeacf4768 (table test-table [id=7b878398c8fc47e3bfb6f5b449170569]) successfully deleted
I20260812 16:48:55.580634 24842 ts_tablet_manager.cc:1918] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:48:55.580808 24842 log.cc:1196] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/386c626a6cc94c8a90fff97eeacf4768
I20260812 16:48:55.581338 24842 ts_tablet_manager.cc:1939] T 386c626a6cc94c8a90fff97eeacf4768 P d8df6f3d96ae45a59f26d124dddc396f: Deleting consensus metadata
I20260812 16:48:55.583295 23933 catalog_manager.cc:4841] TS d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195): tablet 386c626a6cc94c8a90fff97eeacf4768 (table test-table [id=7b878398c8fc47e3bfb6f5b449170569]) successfully deleted
I20260812 16:48:55.592149 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.543886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.28.243.1:35657)
0812 16:48:55.544153 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.544161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.554435 (+ 10274us) server_negotiation.cc:407] Connection header received
0812 16:48:55.554647 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.554655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.554772 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.554940 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:55.556887 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.558433 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.561723 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.562506 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.562639 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.582550 (+ 19911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:55.582570 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:55.582596 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:55.582666 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:55.587421 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:55.588884 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:55.588896 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:55.588899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:55.589258 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:55.589827 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:55.589832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:55.589835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:55.590357 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:55.590644 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.591506 (+ 862us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.591807 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 16:48:55.592850 24265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:55.705767 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 24062
I20260812 16:48:55.745200 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 24287
Aug 12 16:48:55 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:48:55.766599 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.746060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.28.243.2:41195)
0812 16:48:55.746394 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.746401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.746594 (+ 193us) server_negotiation.cc:407] Connection header received
0812 16:48:55.747103 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.747110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.747226 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.747360 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:55.749563 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.750771 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.752718 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.753156 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.753297 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.758323 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:55.758342 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:55.758360 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:55.758407 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:55.761488 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:55.762316 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:55.762324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:55.762326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:55.762615 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:55.763770 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:55.763776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:55.763779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:55.764724 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:55.765425 (+ 701us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.765836 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.766172 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:48:55.768081 24502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:55.869449 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 24287
I20260812 16:48:55.895944 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 24522
Aug 12 16:48:55 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:48:55.915706 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:48:55.896786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.28.243.3:40495)
0812 16:48:55.897283 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 16:48:55.897287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:48:55.897303 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:48:55.897358 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:48:55.897362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:48:55.897426 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:48:55.897524 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:48:55.898966 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.899706 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.901401 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:48:55.901834 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:48:55.902017 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:48:55.908351 (+ 6334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:48:55.908380 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:48:55.908408 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:48:55.908484 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:48:55.912443 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:55.913378 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:55.913386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:55.913389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:55.913542 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:48:55.913936 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:48:55.913940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:48:55.913942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:48:55.914354 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:48:55.914657 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 16:48:55.915060 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 16:48:55.915317 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260812 16:48:55.916379 24762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:56.016702 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 24522
I20260812 16:48:56.040760 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 23897
I20260812 16:48:56.042462 24006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:48:56.168406 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 23897
I20260812 16:49:16.210270 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38365
--webserver_interface=127.28.243.62
--webserver_port=37731
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:38365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:49:16.669998 24860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:16.670338 24860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:16.670477 24860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:16.678123 24860 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 16:49:16.678256 24860 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 16:49:16.678296 24860 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 16:49:16.678352 24860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:49:16.678428 24860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:16.678527 24860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:16.678591 24860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:49:16.678670 24860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:49:16.691737 24860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:38365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:16.693652 24860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:16.695320 24860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:16.728266 24877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:16.728456 24872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:16.728668 24875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:16.729125 24860 server_base.cc:1034] running on GCE node
I20260812 16:49:16.730696 24860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:16.733317 24860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:16.734723 24860 hybrid_clock.cc:648] HybridClock initialized: now 1786553356734621 us; error 57 us; skew 500 ppm
I20260812 16:49:16.735301 24860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:16 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553356, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:16.747229 24860 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:49:16.749949 24860 webserver.cc:460] Webserver started at http://127.28.243.62:37731/ using document root <none> and password file <none>
I20260812 16:49:16.751524 24860 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:16.751744 24860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:16.759963 24860 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:49:16.763859 24887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:16.766351 24860 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:49:16.766606 24860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb"
format_stamp: "Formatted at 2026-08-12 16:48:52 on dist-test-slave-1vmg"
server_key: "eb0936d87d1e1ea0da8ccde2f0d3c779"
server_key_iv: "0146cff4ba2086e1a7ab9d33eba125e3"
server_key_version: "encryptionkey@0"
I20260812 16:49:16.768374 24860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:16.792826 24860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:16.799130 24860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:16.799495 24860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:16.823559 24860 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:38365
I20260812 16:49:16.823597 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:38365 every 8 connection(s)
I20260812 16:49:16.825922 24860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:49:16.831375 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 24860
I20260812 16:49:16.833096 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:35657
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:38365
--webserver_port=44805
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:49:16.837685 24982 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:49:16.843632 24982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap starting.
I20260812 16:49:16.873314 24982 log.cc:824] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Log is configured to *not* fsync() on all Append() calls
I20260812 16:49:16.888275 24982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:49:16.888914 24982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap complete.
I20260812 16:49:16.899418 24982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:16.900441 24982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fcdfc82df4e4ba8b733f5d7fd5223eb, State: Initialized, Role: FOLLOWER
I20260812 16:49:16.901207 24982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:16.901475 24982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:49:16.901602 24982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:49:16.901764 24982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:49:16.906407 24982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:16.907088 24982 leader_election.cc:302] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fcdfc82df4e4ba8b733f5d7fd5223eb; no voters:
I20260812 16:49:16.907802 24982 leader_election.cc:288] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:49:16.907958 24990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:49:16.909132 24990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 LEADER]: Becoming Leader. State: Replica: 7fcdfc82df4e4ba8b733f5d7fd5223eb, State: Running, Role: LEADER
I20260812 16:49:16.909876 24990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:16.910454 24982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:49:16.912887 24996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fcdfc82df4e4ba8b733f5d7fd5223eb. Latest consensus state: current_term: 2 leader_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } } }
I20260812 16:49:16.913756 24996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: This master's current role is: LEADER
I20260812 16:49:16.912916 24991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } } }
I20260812 16:49:16.914341 24991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: This master's current role is: LEADER
I20260812 16:49:16.924749 25008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:49:16.933800 25008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:49:16.935652 25008 catalog_manager.cc:1259] Loaded cluster ID: 8c8c2f5a180948078a4661dccc53a6dc
I20260812 16:49:16.935842 25008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:49:16.941479 25008 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:49:16.943666 25008 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Loaded TSK: 0
I20260812 16:49:16.968493 25008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Generated new TSK 1
I20260812 16:49:16.968941 25008 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:49:17.307092 24986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:17.307598 24986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:17.307771 24986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:17.317828 24986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:17.317999 24986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:17.318176 24986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:49:17.334144 24986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:35657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=44805
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:17.335721 24986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:17.337198 24986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:17.356487 25032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:17.356477 25031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:17.357298 24986 server_base.cc:1034] running on GCE node
W20260812 16:49:17.359251 25037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:17.360250 24986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:17.361753 24986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:17.363234 24986 hybrid_clock.cc:648] HybridClock initialized: now 1786553357363168 us; error 67 us; skew 500 ppm
I20260812 16:49:17.363750 24986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:17 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553357, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:17.368363 24986 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:49:17.370440 24986 webserver.cc:460] Webserver started at http://127.28.243.1:44805/ using document root <none> and password file <none>
I20260812 16:49:17.371585 24986 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:17.371790 24986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:17.380349 24986 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.002s
I20260812 16:49:17.384620 25047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:17.386504 24986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:49:17.386740 24986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4"
format_stamp: "Formatted at 2026-08-12 16:48:53 on dist-test-slave-1vmg"
server_key: "f88327d7a9f0d1d4dfde559aa69c8de3"
server_key_iv: "61a2f433db6180715ed8dde530230d0b"
server_key_version: "encryptionkey@0"
I20260812 16:49:17.388136 24986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:17.412472 24986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:17.418486 24986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:17.418825 24986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:17.420343 24986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:49:17.422245 24986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:49:17.422381 24986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:17.422514 24986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:49:17.422598 24986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:17.461444 24986 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:35657
I20260812 16:49:17.461529 25270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:35657 every 8 connection(s)
I20260812 16:49:17.463132 24986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:49:17.471819 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 24986
I20260812 16:49:17.472709 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:41195
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:38365
--webserver_port=34073
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:49:17 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553357, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:49:17.494823 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:17.467171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:42267 (local address 127.28.243.62:38365)
0812 16:49:17.467880 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:17.467893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:17.468641 (+ 748us) server_negotiation.cc:407] Connection header received
0812 16:49:17.471286 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:17.471305 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:17.471832 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:17.472263 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:17.474034 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:17.475923 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:17.478436 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:17.480291 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:17.480692 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:17.485295 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:17.485305 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:17.485323 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:17.485379 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:17.488913 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:17.489973 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:17.489982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:17.489989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:17.490197 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:17.490732 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:17.490736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:17.490739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:17.491372 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:17.491536 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:17.492610 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0812 16:49:17.493254 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":395,"threads_started":1}
I20260812 16:49:17.494969 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:17.467544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:42267)
0812 16:49:17.468434 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:17.468478 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:17.470611 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:17.472425 (+ 1814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:17.472433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:17.473154 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:49:17.473800 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:17.473821 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:17.476168 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:17.476173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:17.477879 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:17.477899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:17.481091 (+ 3192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:17.481117 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:17.481500 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:17.482428 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:49:17.482454 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:49:17.484935 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:49:17.489163 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:49:17.489172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:49:17.489187 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:49:17.489781 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:49:17.490337 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:49:17.490342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:49:17.490346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:49:17.490606 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:49:17.491548 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:49:17.491557 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:49:17.492045 (+ 488us) client_negotiation.cc:769] Sending connection context
0812 16:49:17.492407 (+ 362us) client_negotiation.cc:240] Negotiation successful
0812 16:49:17.492993 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":331,"threads_started":1}
I20260812 16:49:17.498092 25271 heartbeater.cc:347] Connected to a master server at 127.28.243.62:38365
I20260812 16:49:17.498458 25271 heartbeater.cc:464] Registering TS with master...
I20260812 16:49:17.499293 25271 heartbeater.cc:511] Master 127.28.243.62:38365 requested a full tablet report, sending...
I20260812 16:49:17.501600 24915 ts_manager.cc:194] Registered new tserver with Master: 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1:35657)
I20260812 16:49:17.503178 24915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:42267
W20260812 16:49:17.873652 25279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:17.874102 25279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:17.874284 25279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:17.882112 25279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:17.882261 25279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:17.882421 25279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:49:17.893620 25279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:41195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=34073
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:17.895843 25279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:17.897396 25279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:17.912433 25285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:17.913121 25286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:17.913642 25290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:17.914536 25279 server_base.cc:1034] running on GCE node
I20260812 16:49:17.915293 25279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:17.916424 25279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:17.917696 25279 hybrid_clock.cc:648] HybridClock initialized: now 1786553357917636 us; error 51 us; skew 500 ppm
I20260812 16:49:17.918077 25279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:17 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553357, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:17.921823 25279 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:49:17.923920 25279 webserver.cc:460] Webserver started at http://127.28.243.2:34073/ using document root <none> and password file <none>
I20260812 16:49:17.924746 25279 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:17.924997 25279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:17.932703 25279 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:49:17.936883 25301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:17.938869 25279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:49:17.939154 25279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "d8df6f3d96ae45a59f26d124dddc396f"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "bd2c781ec463cafd69dfcea6f251e959"
server_key_iv: "573f815951845060de38a7db72441ee2"
server_key_version: "encryptionkey@0"
I20260812 16:49:17.940305 25279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:17.969784 25279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:17.975888 25279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:17.976174 25279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:17.977437 25279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:49:17.979120 25279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:49:17.979252 25279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:17.979338 25279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:49:17.979398 25279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:18.019500 25279 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:41195
I20260812 16:49:18.019598 25510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:41195 every 8 connection(s)
I20260812 16:49:18.021075 25279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:49:18.025413 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 25279
I20260812 16:49:18.026297 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:40495
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:38365
--webserver_port=33717
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:49:18 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553357, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:49:18.048494 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.025078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:40409)
0812 16:49:18.026109 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.026161 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.028657 (+ 2496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.029914 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.029927 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.030696 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:49:18.031795 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.031821 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.033350 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.033356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.034773 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.034791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.035844 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.035853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.036184 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.037044 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:49:18.037077 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:49:18.039991 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:49:18.044039 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:49:18.044048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:49:18.044065 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:49:18.044658 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:49:18.045088 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:49:18.045092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:49:18.045094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:49:18.045316 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:49:18.046234 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:49:18.046240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:49:18.046613 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.046865 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.047374 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":366,"threads_started":1}
I20260812 16:49:18.048609 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.024381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:40409 (local address 127.28.243.62:38365)
0812 16:49:18.024955 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.024960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.026298 (+ 1338us) server_negotiation.cc:407] Connection header received
0812 16:49:18.029205 (+ 2907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.029212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.029348 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.029493 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:18.032316 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.033133 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.034995 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.035551 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.035680 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.040309 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:18.040315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:18.040318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:18.040360 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:18.043836 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.044802 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.044808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.044811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.044969 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.045509 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.045515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.045518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.045948 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:18.046272 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.047255 (+ 983us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.048048 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":296,"threads_started":1}
I20260812 16:49:18.050173 25511 heartbeater.cc:347] Connected to a master server at 127.28.243.62:38365
I20260812 16:49:18.050493 25511 heartbeater.cc:464] Registering TS with master...
I20260812 16:49:18.051324 25511 heartbeater.cc:511] Master 127.28.243.62:38365 requested a full tablet report, sending...
I20260812 16:49:18.053117 24915 ts_manager.cc:194] Registered new tserver with Master: d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195)
I20260812 16:49:18.053999 24915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:40409
W20260812 16:49:18.459489 25518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:18.459816 25518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:18.459931 25518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:18.472098 25518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:18.472252 25518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:18.472393 25518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:49:18.484349 25518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:40495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=33717
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:18.486979 25518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:18.489286 25518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 16:49:18.508800 25271 heartbeater.cc:503] Master 127.28.243.62:38365 was elected leader, sending a full tablet report...
W20260812 16:49:18.508930 25529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:18.509321 25518 server_base.cc:1034] running on GCE node
W20260812 16:49:18.510531 25530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:18.510550 25534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:18.512311 25518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:18.513774 25518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:18.515079 25518 hybrid_clock.cc:648] HybridClock initialized: now 1786553358514986 us; error 92 us; skew 500 ppm
I20260812 16:49:18.515501 25518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:18 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553358, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:18.519232 25518 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:49:18.521067 25518 webserver.cc:460] Webserver started at http://127.28.243.3:33717/ using document root <none> and password file <none>
I20260812 16:49:18.521880 25518 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:18.522038 25518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:18.530663 25518 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:49:18.535326 25545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:18.536934 25518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:49:18.537122 25518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "56c351cc8fc2413691f1f3bb42dcf454"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "d5a195af1442eb7247e72e36d4f10196"
server_key_iv: "bffb9c08d6ddb7bd2ef6d5a20f74b47c"
server_key_version: "encryptionkey@0"
I20260812 16:49:18.538208 25518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:18.565069 25518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:18.569700 25518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:18.570031 25518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:18.571420 25518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:49:18.573742 25518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:49:18.573879 25518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:18.573998 25518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:49:18.574081 25518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:18.618429 25518 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:40495
I20260812 16:49:18.618481 25741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:40495 every 8 connection(s)
I20260812 16:49:18.620088 25518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:49:18.629590 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 25518
Aug 12 16:49:18 dist-test-slave-1vmg krb5kdc[23876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553358, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:49:18.648388 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.623031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:48285 (local address 127.28.243.62:38365)
0812 16:49:18.623549 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.623555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.626060 (+ 2505us) server_negotiation.cc:407] Connection header received
0812 16:49:18.628247 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.628253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.628349 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.628458 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:18.630810 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.632397 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.634745 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.635370 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.635521 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.639639 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:18.639644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:18.639647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:18.639683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:18.643264 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.644906 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.644910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.644913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.645061 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.645751 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.645758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.645762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.646112 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:18.646245 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.647759 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.648045 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":407,"thread_start_us":241,"threads_started":1}
I20260812 16:49:18.648407 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.630747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.28.243.62:38365)
0812 16:49:18.631460 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.631466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.631854 (+ 388us) server_negotiation.cc:407] Connection header received
0812 16:49:18.632042 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.632047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.632122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.632222 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:18.634380 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.635501 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.637096 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.637521 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.637651 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.640757 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:18.640763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:18.640766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:18.640810 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:18.644069 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.645002 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.645009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.645012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.645316 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.646265 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.646270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.646273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.646538 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:18.646748 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.647645 (+ 897us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.647913 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":551,"spinlock_wait_cycles":2944,"thread_start_us":346,"threads_started":1}
I20260812 16:49:18.651628 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.624436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:48285)
0812 16:49:18.625849 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.625914 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.627889 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.628770 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.628779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.629571 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:49:18.630463 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.630487 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.632720 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.632725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.634420 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.634445 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.635729 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.635740 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.636024 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.636766 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:49:18.636793 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:49:18.639371 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:49:18.643505 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:49:18.643514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:49:18.643532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:49:18.644081 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:49:18.645214 (+ 1133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:49:18.645218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:49:18.645220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:49:18.645527 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:49:18.646301 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:49:18.646311 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:49:18.646900 (+ 589us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.647687 (+ 787us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.648829 (+ 1142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":527,"threads_started":1}
I20260812 16:49:18.653707 25742 heartbeater.cc:347] Connected to a master server at 127.28.243.62:38365
I20260812 16:49:18.654088 25742 heartbeater.cc:464] Registering TS with master...
I20260812 16:49:18.654978 25742 heartbeater.cc:511] Master 127.28.243.62:38365 requested a full tablet report, sending...
I20260812 16:49:18.656683 24915 ts_manager.cc:194] Registered new tserver with Master: 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3:40495)
I20260812 16:49:18.657550 24915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:48285
I20260812 16:49:18.665536 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:49:18.675048 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.667026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.28.243.62:38365)
0812 16:49:18.667285 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.667292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.667841 (+ 549us) server_negotiation.cc:407] Connection header received
0812 16:49:18.668165 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.668170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.668380 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.668557 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:49:18.669148 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.669953 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.672177 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.672709 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.672843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.673915 (+ 1072us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:49:18.674231 (+ 316us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 16:49:18.674726 (+ 495us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32838: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 16:49:18.675200 25752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32838: token expired
W20260812 16:49:18.676091 24777 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.243.62:38365)
I20260812 16:49:18.692135 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.676750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.28.243.62:38365)
0812 16:49:18.676908 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.676913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.676930 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:49:18.677055 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.677058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.677122 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.677203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:18.678726 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.679574 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.681200 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.681664 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.681778 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.685318 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:18.685324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:18.685328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:18.685375 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:18.688711 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.689662 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.689666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.689669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.689886 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.690392 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.690398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.690401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.690878 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:18.691195 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.691532 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.691803 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:49:18.696666 24915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:49:18.704876 24915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:49:18.734679 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.719895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:40495 (local address 127.0.0.1:60940)
0812 16:49:18.720980 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.721053 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.721304 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.725290 (+ 3986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.725297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.725333 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:49:18.725862 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.725877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.728246 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.728253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.730890 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.730943 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.733341 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.733349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.733625 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.733694 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.733874 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.734117 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":53,"thread_start_us":565,"threads_started":1}
I20260812 16:49:18.737142 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.719490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.28.243.3:40495)
0812 16:49:18.723548 (+ 4058us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.723557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.723596 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:49:18.723719 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.723727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.724394 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.724874 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:49:18.726133 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.728018 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.731191 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.732999 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.733169 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.733476 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.735044 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.735302 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3868,"thread_start_us":3696,"threads_started":1}
I20260812 16:49:18.738415 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.718621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35657 (local address 127.0.0.1:38264)
0812 16:49:18.720980 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.721049 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.721310 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.722455 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.722464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.722497 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:49:18.722986 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.723000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.726809 (+ 3809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.726815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.734933 (+ 8118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.734953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.737545 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.737553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.737738 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.737771 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.737854 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.737993 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2074,"mutex_wait_us":5117,"thread_start_us":1759,"threads_started":1}
I20260812 16:49:18.739256 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.718820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.28.243.1:35657)
0812 16:49:18.720048 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.720059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.721223 (+ 1164us) server_negotiation.cc:407] Connection header received
0812 16:49:18.721618 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.721627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.721912 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.722129 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:49:18.723197 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.725132 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.735180 (+ 10048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.736933 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.737091 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.737354 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.738721 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.738910 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":810,"threads_started":1}
I20260812 16:49:18.741648 25622 tablet_service.cc:1467] Processing CreateTablet for tablet 240b495ea11c4f898ec2f70765ab66d3 (DEFAULT_TABLE table=test-table [id=d203f7acba4e436ca93dbc97dbb27aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:49:18.743356 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240b495ea11c4f898ec2f70765ab66d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:49:18.743870 25136 tablet_service.cc:1467] Processing CreateTablet for tablet 240b495ea11c4f898ec2f70765ab66d3 (DEFAULT_TABLE table=test-table [id=d203f7acba4e436ca93dbc97dbb27aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:49:18.745253 25136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240b495ea11c4f898ec2f70765ab66d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:49:18.754755 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.721493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:41195 (local address 127.0.0.1:57366)
0812 16:49:18.737496 (+ 16003us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.737524 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.737651 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.739295 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.739301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.739328 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:49:18.739497 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.739508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.749536 (+ 10028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.749545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.751674 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.751697 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.753916 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.753924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.754123 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.754161 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.754267 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.754430 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13401,"thread_start_us":18439,"threads_started":1}
I20260812 16:49:18.755045 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.721546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.28.243.2:41195)
0812 16:49:18.738272 (+ 16726us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.738282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.738316 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:49:18.738442 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.738452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.738719 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.738976 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:49:18.739689 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.740926 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.751939 (+ 11013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.753659 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.753864 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.754110 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.754367 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.754593 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16508,"thread_start_us":17962,"threads_started":1}
I20260812 16:49:18.759903 25389 tablet_service.cc:1467] Processing CreateTablet for tablet 240b495ea11c4f898ec2f70765ab66d3 (DEFAULT_TABLE table=test-table [id=d203f7acba4e436ca93dbc97dbb27aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:49:18.761984 25389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240b495ea11c4f898ec2f70765ab66d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:49:18.761945 25771 tablet_bootstrap.cc:492] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Bootstrap starting.
I20260812 16:49:18.764497 25770 tablet_bootstrap.cc:492] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Bootstrap starting.
I20260812 16:49:18.766497 25771 tablet_bootstrap.cc:654] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Neither blocks nor log segments found. Creating new log.
I20260812 16:49:18.767792 25771 log.cc:824] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:49:18.768435 25770 tablet_bootstrap.cc:654] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Neither blocks nor log segments found. Creating new log.
I20260812 16:49:18.769779 25770 log.cc:824] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Log is configured to *not* fsync() on all Append() calls
I20260812 16:49:18.771749 25771 tablet_bootstrap.cc:492] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: No bootstrap required, opened a new log
I20260812 16:49:18.772089 25771 ts_tablet_manager.cc:1397] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260812 16:49:18.775565 25770 tablet_bootstrap.cc:492] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: No bootstrap required, opened a new log
I20260812 16:49:18.775923 25770 ts_tablet_manager.cc:1397] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260812 16:49:18.778405 25778 tablet_bootstrap.cc:492] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Bootstrap starting.
I20260812 16:49:18.781796 25778 tablet_bootstrap.cc:654] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Neither blocks nor log segments found. Creating new log.
I20260812 16:49:18.782905 25778 log.cc:824] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:49:18.783934 25771 raft_consensus.cc:348] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.784458 25771 raft_consensus.cc:374] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:49:18.784593 25771 raft_consensus.cc:729] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46986aa1aa3d4e40a9279dc6d5a9ecf4, State: Initialized, Role: FOLLOWER
I20260812 16:49:18.785399 25771 consensus_queue.cc:260] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.786659 25778 tablet_bootstrap.cc:492] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: No bootstrap required, opened a new log
I20260812 16:49:18.786993 25778 ts_tablet_manager.cc:1397] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260812 16:49:18.787569 25771 ts_tablet_manager.cc:1428] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260812 16:49:18.787317 25770 raft_consensus.cc:348] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.787822 25770 raft_consensus.cc:374] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:49:18.787932 25770 raft_consensus.cc:729] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c351cc8fc2413691f1f3bb42dcf454, State: Initialized, Role: FOLLOWER
I20260812 16:49:18.788610 25770 consensus_queue.cc:260] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.790366 25742 heartbeater.cc:503] Master 127.28.243.62:38365 was elected leader, sending a full tablet report...
I20260812 16:49:18.791199 25770 ts_tablet_manager.cc:1428] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260812 16:49:18.797528 25778 raft_consensus.cc:348] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.798273 25778 raft_consensus.cc:374] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:49:18.798409 25778 raft_consensus.cc:729] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8df6f3d96ae45a59f26d124dddc396f, State: Initialized, Role: FOLLOWER
I20260812 16:49:18.799111 25778 consensus_queue.cc:260] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.800594 25511 heartbeater.cc:503] Master 127.28.243.62:38365 was elected leader, sending a full tablet report...
I20260812 16:49:18.800951 25778 ts_tablet_manager.cc:1428] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260812 16:49:18.809213 25784 raft_consensus.cc:482] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:49:18.809605 25784 raft_consensus.cc:504] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.812086 25784 leader_election.cc:288] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1:35657), d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195)
I20260812 16:49:18.820294 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.812587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:35657 (local address 127.28.243.3:49293)
0812 16:49:18.812851 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.812873 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.813048 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.813697 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.813703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.813744 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:49:18.814016 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.814032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.815856 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.815860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.817726 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.817746 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.819604 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.819608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.819782 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.819809 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.819890 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.820000 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":19}
I20260812 16:49:18.820359 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.812621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:49293 (local address 127.28.243.1:35657)
0812 16:49:18.812955 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.812962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.812993 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:49:18.813241 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.813246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.813339 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.813456 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:49:18.814240 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.815671 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.818025 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.819402 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.819611 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.819701 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.819946 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.820104 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 16:49:18.821324 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.813346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:50301 (local address 127.28.243.2:41195)
0812 16:49:18.813571 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.813578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.814219 (+ 641us) server_negotiation.cc:407] Connection header received
0812 16:49:18.814327 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.814333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.814427 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.814629 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:49:18.815382 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.816503 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.818683 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.820105 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.820274 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.820349 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.820921 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.821071 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:49:18.821741 25175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b495ea11c4f898ec2f70765ab66d3" candidate_uuid: "56c351cc8fc2413691f1f3bb42dcf454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" is_pre_election: true
I20260812 16:49:18.821995 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.813123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:41195 (local address 127.28.243.3:50301)
0812 16:49:18.814057 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:18.814083 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:49:18.814217 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:49:18.814682 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:49:18.814687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:49:18.814721 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:49:18.815170 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.815183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.816671 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.816677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.818448 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:49:18.818460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.820526 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.820531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:49:18.820712 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.820745 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:49:18.821527 (+ 782us) client_negotiation.cc:240] Negotiation successful
0812 16:49:18.821682 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":30,"thread_start_us":482,"threads_started":1}
I20260812 16:49:18.822395 25175 raft_consensus.cc:2454] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c351cc8fc2413691f1f3bb42dcf454 in term 0.
I20260812 16:49:18.823185 25550 leader_election.cc:302] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46986aa1aa3d4e40a9279dc6d5a9ecf4, 56c351cc8fc2413691f1f3bb42dcf454; no voters:
I20260812 16:49:18.823143 25432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b495ea11c4f898ec2f70765ab66d3" candidate_uuid: "56c351cc8fc2413691f1f3bb42dcf454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df6f3d96ae45a59f26d124dddc396f" is_pre_election: true
I20260812 16:49:18.823596 25784 raft_consensus.cc:2781] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:49:18.823696 25432 raft_consensus.cc:2454] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c351cc8fc2413691f1f3bb42dcf454 in term 0.
I20260812 16:49:18.823753 25784 raft_consensus.cc:482] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:49:18.823881 25784 raft_consensus.cc:3037] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:49:18.827243 25784 raft_consensus.cc:504] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.828513 25784 leader_election.cc:288] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [CANDIDATE]: Term 1 election: Requested vote from peers 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1:35657), d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195)
I20260812 16:49:18.829353 25175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b495ea11c4f898ec2f70765ab66d3" candidate_uuid: "56c351cc8fc2413691f1f3bb42dcf454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4"
I20260812 16:49:18.829690 25175 raft_consensus.cc:3037] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:49:18.829890 25432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b495ea11c4f898ec2f70765ab66d3" candidate_uuid: "56c351cc8fc2413691f1f3bb42dcf454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df6f3d96ae45a59f26d124dddc396f"
I20260812 16:49:18.830184 25432 raft_consensus.cc:3037] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:49:18.834208 25175 raft_consensus.cc:2454] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c351cc8fc2413691f1f3bb42dcf454 in term 1.
I20260812 16:49:18.834306 25432 raft_consensus.cc:2454] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c351cc8fc2413691f1f3bb42dcf454 in term 1.
I20260812 16:49:18.834847 25550 leader_election.cc:302] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46986aa1aa3d4e40a9279dc6d5a9ecf4, 56c351cc8fc2413691f1f3bb42dcf454; no voters:
I20260812 16:49:18.835258 25784 raft_consensus.cc:2781] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:49:18.835800 25784 raft_consensus.cc:686] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 LEADER]: Becoming Leader. State: Replica: 56c351cc8fc2413691f1f3bb42dcf454, State: Running, Role: LEADER
I20260812 16:49:18.836431 25784 consensus_queue.cc:237] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } }
I20260812 16:49:18.842684 24913 catalog_manager.cc:5495] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3). New cstate: current_term: 1 leader_uuid: "56c351cc8fc2413691f1f3bb42dcf454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c351cc8fc2413691f1f3bb42dcf454" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 40495 } health_report { overall_health: HEALTHY } } }
W20260812 16:49:18.872041 25745 tablet.cc:2369] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:49:18.894860 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.887128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.28.243.3:40495)
0812 16:49:18.887365 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.887370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.887397 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:49:18.887549 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.887554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.887664 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.887870 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:49:18.888595 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.889724 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.891723 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.892217 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.892487 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.893172 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:49:18.893843 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:49:18.893972 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.894503 (+ 531us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.894598 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:49:18.905464 25432 raft_consensus.cc:1261] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Refusing update from remote peer 56c351cc8fc2413691f1f3bb42dcf454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:49:18.905464 25175 raft_consensus.cc:1261] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Refusing update from remote peer 56c351cc8fc2413691f1f3bb42dcf454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:49:18.906853 25787 consensus_queue.cc:1035] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8df6f3d96ae45a59f26d124dddc396f" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:49:18.907409 25784 consensus_queue.cc:1035] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:49:18.923759 25793 mvcc.cc:205] Tried to move back new op lower bound from 7317722558060621824 to 7317722557798690816. Current Snapshot: MvccSnapshot[applied={T|T < 7317722558060621824}]
I20260812 16:49:18.942700 24915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:49:18.943089 24915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:49:18.949276 24915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Sending DeleteTablet for 3 replicas of tablet 240b495ea11c4f898ec2f70765ab66d3
I20260812 16:49:18.952158 25136 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b495ea11c4f898ec2f70765ab66d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:49:18 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:38264
I20260812 16:49:18.952961 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 24986
I20260812 16:49:18.952831 25389 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b495ea11c4f898ec2f70765ab66d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:49:18 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260812 16:49:18.954509 25622 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b495ea11c4f898ec2f70765ab66d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:49:18 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:60940
I20260812 16:49:18.966238 25802 tablet_replica.cc:330] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: stopping tablet replica
I20260812 16:49:18.967799 25808 tablet_replica.cc:330] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: stopping tablet replica
I20260812 16:49:18.967788 25802 raft_consensus.cc:2229] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:49:18.970469 25802 raft_consensus.cc:2258] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:49:18.972528 25808 raft_consensus.cc:2229] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:49:18.973196 25808 raft_consensus.cc:2258] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:49:18.974283 25802 ts_tablet_manager.cc:1905] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:49:18.978243 25808 ts_tablet_manager.cc:1905] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:49:18.982995 25802 ts_tablet_manager.cc:1918] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:49:18.983398 25802 log.cc:1196] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/240b495ea11c4f898ec2f70765ab66d3
I20260812 16:49:18.984205 25802 ts_tablet_manager.cc:1939] T 240b495ea11c4f898ec2f70765ab66d3 P 46986aa1aa3d4e40a9279dc6d5a9ecf4: Deleting consensus metadata
I20260812 16:49:18.987684 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:18.955424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.28.243.1:35657)
0812 16:49:18.955662 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:18.955669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:18.955694 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:49:18.955781 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:18.955787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:18.955932 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:18.956066 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:18.958621 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.960242 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.963080 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:18.965781 (+ 2701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:18.967806 (+ 2025us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:18.975832 (+ 8026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:18.975852 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:18.975871 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:18.975924 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:18.980351 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.981831 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.981844 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.981848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.982086 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:18.982693 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:18.982699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:18.982703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:18.983829 (+ 1126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:18.986324 (+ 2495us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:18.986498 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 16:49:18.987274 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:49:18.987778 24895 catalog_manager.cc:4841] TS 46986aa1aa3d4e40a9279dc6d5a9ecf4 (127.28.243.1:35657): tablet 240b495ea11c4f898ec2f70765ab66d3 (table test-table [id=d203f7acba4e436ca93dbc97dbb27aa4]) successfully deleted
I20260812 16:49:18.989077 25809 tablet_replica.cc:330] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: stopping tablet replica
I20260812 16:49:18.990408 25263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:18.990826 25809 raft_consensus.cc:2229] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:49:18.992072 25809 raft_consensus.cc:2258] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:49:18.993468 25808 ts_tablet_manager.cc:1918] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:49:18.993747 25808 log.cc:1196] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/240b495ea11c4f898ec2f70765ab66d3
I20260812 16:49:18.994899 25808 ts_tablet_manager.cc:1939] T 240b495ea11c4f898ec2f70765ab66d3 P d8df6f3d96ae45a59f26d124dddc396f: Deleting consensus metadata
I20260812 16:49:18.996104 25809 ts_tablet_manager.cc:1905] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:49:18.997340 24895 catalog_manager.cc:4841] TS d8df6f3d96ae45a59f26d124dddc396f (127.28.243.2:41195): tablet 240b495ea11c4f898ec2f70765ab66d3 (table test-table [id=d203f7acba4e436ca93dbc97dbb27aa4]) successfully deleted
I20260812 16:49:19.005333 25809 ts_tablet_manager.cc:1918] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:49:19.005792 25809 log.cc:1196] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/240b495ea11c4f898ec2f70765ab66d3
I20260812 16:49:19.007305 25809 ts_tablet_manager.cc:1939] T 240b495ea11c4f898ec2f70765ab66d3 P 56c351cc8fc2413691f1f3bb42dcf454: Deleting consensus metadata
I20260812 16:49:19.010831 24889 catalog_manager.cc:4841] TS 56c351cc8fc2413691f1f3bb42dcf454 (127.28.243.3:40495): tablet 240b495ea11c4f898ec2f70765ab66d3 (table test-table [id=d203f7acba4e436ca93dbc97dbb27aa4]) successfully deleted
I20260812 16:49:19.105305 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 24986
I20260812 16:49:19.149959 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 25279
I20260812 16:49:19.170290 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:19.151664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.28.243.2:41195)
0812 16:49:19.151888 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:19.151895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:19.152091 (+ 196us) server_negotiation.cc:407] Connection header received
0812 16:49:19.152233 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:19.152241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:19.152341 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:19.152462 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:19.154061 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:19.154850 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:19.156356 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:19.157082 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:19.157256 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:19.160461 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:19.160488 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:19.160527 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:19.160585 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:19.165718 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:19.166827 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:19.166845 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:19.166849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:19.167248 (+ 399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:19.167957 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:19.167965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:19.167969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:19.168557 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:19.168918 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:19.169556 (+ 638us) server_negotiation.cc:299] Negotiation successful
0812 16:49:19.169872 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:49:19.171355 25502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:19.268654 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 25279
I20260812 16:49:19.304138 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 25518
I20260812 16:49:19.322806 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:19.305447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.28.243.3:40495)
0812 16:49:19.305694 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:19.305700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:19.305725 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:49:19.305934 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:19.305939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:19.306045 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:19.306141 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:19.308287 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:19.309231 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:19.310811 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:19.311646 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:19.311813 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:19.315366 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:19.315388 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:19.315408 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:19.315459 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:19.318777 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:19.319975 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:19.319984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:19.319986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:19.320173 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:19.320648 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:19.320653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:19.320655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:19.321132 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:19.321454 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:19.321847 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 16:49:19.322140 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:49:19.324110 25735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:19.470532 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 25518
I20260812 16:49:19.502170 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 24860
I20260812 16:49:19.503911 24975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:19.640964 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 24860
I20260812 16:49:39.685662 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38365
--webserver_interface=127.28.243.62
--webserver_port=37731
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:38365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:49:40.122419 25817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:40.122776 25817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:40.122874 25817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:40.135355 25817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 16:49:40.135502 25817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 16:49:40.135541 25817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 16:49:40.135602 25817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:49:40.135660 25817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:40.135738 25817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:40.135792 25817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:49:40.135847 25817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:49:40.148485 25817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:38365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:40.150426 25817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:40.152741 25817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:40.167411 25824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:40.167434 25827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:40.167460 25823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:40.173388 25817 server_base.cc:1034] running on GCE node
I20260812 16:49:40.174407 25817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:40.176198 25817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:40.177480 25817 hybrid_clock.cc:648] HybridClock initialized: now 1786553380177412 us; error 45 us; skew 500 ppm
I20260812 16:49:40.177871 25817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:40 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553380, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:40.181495 25817 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:49:40.183401 25817 webserver.cc:460] Webserver started at http://127.28.243.62:37731/ using document root <none> and password file <none>
I20260812 16:49:40.184252 25817 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:40.184527 25817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:40.192149 25817 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:49:40.196431 25841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:40.198292 25817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 16:49:40.198539 25817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb"
format_stamp: "Formatted at 2026-08-12 16:48:52 on dist-test-slave-1vmg"
server_key: "eb0936d87d1e1ea0da8ccde2f0d3c779"
server_key_iv: "0146cff4ba2086e1a7ab9d33eba125e3"
server_key_version: "encryptionkey@0"
I20260812 16:49:40.199730 25817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:40.218225 25817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:40.223322 25817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:40.223636 25817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:40.247324 25817 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:38365
I20260812 16:49:40.247339 25934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:38365 every 8 connection(s)
I20260812 16:49:40.250360 25817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:49:40.254375 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 25817
I20260812 16:49:40.255607 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:35657
--local_ip_for_outbound_sockets=127.28.243.1
--tserver_master_addrs=127.28.243.62:38365
--webserver_port=44805
--webserver_interface=127.28.243.1
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:49:40.259918 25935 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:49:40.263794 25935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap starting.
I20260812 16:49:40.291774 25935 log.cc:824] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Log is configured to *not* fsync() on all Append() calls
I20260812 16:49:40.314540 25935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:49:40.315263 25935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Bootstrap complete.
I20260812 16:49:40.324718 25935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:40.325814 25935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fcdfc82df4e4ba8b733f5d7fd5223eb, State: Initialized, Role: FOLLOWER
I20260812 16:49:40.326859 25935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:40.327364 25935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:49:40.327543 25935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:49:40.327786 25935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 2 FOLLOWER]: Advancing to term 3
I20260812 16:49:40.334534 25935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:40.335469 25935 leader_election.cc:302] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fcdfc82df4e4ba8b733f5d7fd5223eb; no voters:
I20260812 16:49:40.336268 25935 leader_election.cc:288] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 16:49:40.336438 25941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 3 FOLLOWER]: Leader election won for term 3
I20260812 16:49:40.337514 25941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [term 3 LEADER]: Becoming Leader. State: Replica: 7fcdfc82df4e4ba8b733f5d7fd5223eb, State: Running, Role: LEADER
I20260812 16:49:40.338248 25941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } }
I20260812 16:49:40.338718 25935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:49:40.341177 25943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fcdfc82df4e4ba8b733f5d7fd5223eb. Latest consensus state: current_term: 3 leader_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } } }
I20260812 16:49:40.341647 25943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: This master's current role is: LEADER
I20260812 16:49:40.341820 25942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcdfc82df4e4ba8b733f5d7fd5223eb" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38365 } } }
I20260812 16:49:40.342192 25942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb [sys.catalog]: This master's current role is: LEADER
I20260812 16:49:40.354524 25958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:49:40.364917 25958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:49:40.367342 25958 catalog_manager.cc:1259] Loaded cluster ID: 8c8c2f5a180948078a4661dccc53a6dc
I20260812 16:49:40.367609 25958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:49:40.372936 25958 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:49:40.375042 25958 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Loaded TSK: 0
I20260812 16:49:40.375376 25958 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7fcdfc82df4e4ba8b733f5d7fd5223eb: Loaded TSK: 1
F20260812 16:49:40.376611 25958 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f07616969ad google::LogMessage::Fail() at ??:0
@ 0x7f076169988a google::LogMessage::SendToLog() at ??:0
@ 0x7f07616964de google::LogMessage::Flush() at ??:0
@ 0x7f0761697a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f07653fa7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f07653e973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f07770c91c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f07770c8783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f07771542a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f07771540c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f077719e5e9 std::function<>::operator()() at ??:0
@ 0x7f07770cccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f07770cac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0777152610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0777152439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f07642e2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f07642e591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f07642e5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f07642c0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f076759a6db start_thread at ??:0
@ 0x7f076053061f clone at ??:0
W20260812 16:49:40.698303 25939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:40.698666 25939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:40.698797 25939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:40.706583 25939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:40.706727 25939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:40.706882 25939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:49:40.717649 25939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:35657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=44805
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:40.719151 25939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:40.720574 25939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:40.735430 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:40.735500 26009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:40.735476 26007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:40.736060 25939 server_base.cc:1034] running on GCE node
I20260812 16:49:40.736956 25939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:40.738080 25939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:40.741528 25939 hybrid_clock.cc:648] HybridClock initialized: now 1786553380741437 us; error 64 us; skew 500 ppm
I20260812 16:49:40.742089 25939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:40 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553380, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:40.746851 25939 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:49:40.749186 25939 webserver.cc:460] Webserver started at http://127.28.243.1:44805/ using document root <none> and password file <none>
I20260812 16:49:40.750034 25939 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:40.750197 25939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:40.758052 25939 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:49:40.762266 26033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:40.764029 25939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 16:49:40.764245 25939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "46986aa1aa3d4e40a9279dc6d5a9ecf4"
format_stamp: "Formatted at 2026-08-12 16:48:53 on dist-test-slave-1vmg"
server_key: "f88327d7a9f0d1d4dfde559aa69c8de3"
server_key_iv: "61a2f433db6180715ed8dde530230d0b"
server_key_version: "encryptionkey@0"
I20260812 16:49:40.765398 25939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:40.783301 25939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:40.788101 25939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:40.788426 25939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:40.789705 25939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:49:40.791426 25939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:49:40.791538 25939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:40.791666 25939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:49:40.791749 25939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:40.830439 25939 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:35657
I20260812 16:49:40.830484 26238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:35657 every 8 connection(s)
I20260812 16:49:40.832099 25939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:49:40.838174 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 25939
I20260812 16:49:40.838676 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:40.835673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:60429)
0812 16:49:40.836855 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:40.837239 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":338,"threads_started":1}
I20260812 16:49:40.839272 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:41195
--local_ip_for_outbound_sockets=127.28.243.2
--tserver_master_addrs=127.28.243.62:38365
--webserver_port=34073
--webserver_interface=127.28.243.2
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:49:40.840090 26239 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:38365 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.62:38365: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
I20260812 16:49:40.841432 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:40.840862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:59223)
0812 16:49:40.841172 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:40.841279 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260812 16:49:40.842541 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:40.842100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:57763)
0812 16:49:40.842274 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:40.842377 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 16:49:40.843030 26239 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:49:41.244597 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:41.245066 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:41.245271 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:41.255934 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:41.256102 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:41.256310 26245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:49:41.270231 26245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:41195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=34073
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:41.272231 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:41.273983 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:41.290844 26251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:41.290977 26254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:41.291246 26252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:41.292439 26245 server_base.cc:1034] running on GCE node
I20260812 16:49:41.293167 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:41.294312 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:41.295573 26245 hybrid_clock.cc:648] HybridClock initialized: now 1786553381295513 us; error 50 us; skew 500 ppm
I20260812 16:49:41.295961 26245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:41 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553381, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:41.300494 26245 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:49:41.302361 26245 webserver.cc:460] Webserver started at http://127.28.243.2:34073/ using document root <none> and password file <none>
I20260812 16:49:41.303226 26245 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:41.303360 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:41.309845 26245 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:49:41.313308 26267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:41.315032 26245 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:49:41.315253 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "d8df6f3d96ae45a59f26d124dddc396f"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "bd2c781ec463cafd69dfcea6f251e959"
server_key_iv: "573f815951845060de38a7db72441ee2"
server_key_version: "encryptionkey@0"
I20260812 16:49:41.316439 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:41.342119 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:41.346872 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:41.347208 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:41.348330 26245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:49:41.349841 26245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:49:41.349958 26245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:41.350068 26245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:49:41.350143 26245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:41.389866 26245 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:41195
I20260812 16:49:41.389928 26477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:41195 every 8 connection(s)
I20260812 16:49:41.391467 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:49:41.396840 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.394728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:59327)
0812 16:49:41.395630 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.395908 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":401,"threads_started":1}
W20260812 16:49:41.397531 26479 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:38365 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.62:38365: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
I20260812 16:49:41.398191 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 26245
I20260812 16:49:41.398944 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.398376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:55505)
0812 16:49:41.398662 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.398770 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:49:41.398995 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:40495
--local_ip_for_outbound_sockets=127.28.243.3
--tserver_master_addrs=127.28.243.62:38365
--webserver_port=33717
--webserver_interface=127.28.243.3
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:49:41.400269 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.399797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:39275)
0812 16:49:41.400000 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.400116 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 16:49:41.400697 26479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:49:41.784837 26482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:41.785176 26482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:41.785308 26482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:41.793208 26482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:41.793370 26482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:41.793533 26482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:49:41.805742 26482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:40495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=33717
--tserver_master_addrs=127.28.243.62:38365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:41.807418 26482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:41.808969 26482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:41.824472 26488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:41.824519 26493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:41.824546 26496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:41.825812 26482 server_base.cc:1034] running on GCE node
I20260812 16:49:41.826538 26482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:41.827785 26482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:41.829082 26482 hybrid_clock.cc:648] HybridClock initialized: now 1786553381828928 us; error 134 us; skew 500 ppm
I20260812 16:49:41.829452 26482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:41 dist-test-slave-1vmg krb5kdc[23876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553381, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:41.832897 26482 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:49:41.834645 26482 webserver.cc:460] Webserver started at http://127.28.243.3:33717/ using document root <none> and password file <none>
I20260812 16:49:41.835477 26482 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:41.835634 26482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:41.842834 26482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 16:49:41.844578 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.843698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:33187)
0812 16:49:41.844230 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.844341 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
I20260812 16:49:41.846169 26509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:41.847873 26482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:49:41.848071 26482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "56c351cc8fc2413691f1f3bb42dcf454"
format_stamp: "Formatted at 2026-08-12 16:48:54 on dist-test-slave-1vmg"
server_key: "d5a195af1442eb7247e72e36d4f10196"
server_key_iv: "bffb9c08d6ddb7bd2ef6d5a20f74b47c"
server_key_version: "encryptionkey@0"
I20260812 16:49:41.849193 26482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:41.865839 26482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:41.870685 26482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:41.871038 26482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:41.872090 26482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:49:41.873541 26482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:49:41.873649 26482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:41.873760 26482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:49:41.873831 26482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:41.912271 26482 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:40495
I20260812 16:49:41.912431 26712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:40495 every 8 connection(s)
I20260812 16:49:41.913906 26482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:49:41.915746 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 26482
I20260812 16:49:41.920775 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.918157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:56523)
0812 16:49:41.919322 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.919634 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":443,"threads_started":1}
W20260812 16:49:41.921669 26713 heartbeater.cc:644] Failed to heartbeat to 127.28.243.62:38365 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.62:38365: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
I20260812 16:49:41.922742 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.922389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:40395)
0812 16:49:41.922543 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.922620 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 16:49:41.923699 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:41.923347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:50089)
0812 16:49:41.923499 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:41.923581 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 16:49:41.923957 26713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 16:49:42.403028 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:42.401809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:33331)
0812 16:49:42.402545 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:42.402676 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":364,"threads_started":1}
I20260812 16:49:42.847291 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:42.846117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:44075)
0812 16:49:42.846798 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:42.846903 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
I20260812 16:49:42.925894 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:42.924644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:51911)
0812 16:49:42.925421 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:42.925584 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":253,"threads_started":1}
I20260812 16:49:43.405246 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:43.404222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:36085)
0812 16:49:43.404747 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:43.404846 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":257,"threads_started":1}
I20260812 16:49:43.850164 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:43.848937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:38171)
0812 16:49:43.849682 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:43.849791 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":478,"threads_started":1}
I20260812 16:49:43.928581 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:43.927459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:48627)
0812 16:49:43.928232 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:43.928350 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":337,"threads_started":1}
I20260812 16:49:44.407820 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:44.406724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:36455)
0812 16:49:44.407451 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:44.407577 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":435,"threads_started":1}
I20260812 16:49:44.852558 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:44.851546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:49173)
0812 16:49:44.852163 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:44.852308 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260812 16:49:44.931185 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:44.930136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:37935)
0812 16:49:44.930774 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:44.930903 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":335,"threads_started":1}
I20260812 16:49:45.410409 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:45.409318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:36739)
0812 16:49:45.410022 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:45.410156 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":356,"threads_started":1}
I20260812 16:49:45.855357 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:45.854330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:39977)
0812 16:49:45.855037 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:45.855148 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":326,"threads_started":1}
I20260812 16:49:45.933833 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:45.932802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:50081)
0812 16:49:45.933459 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:45.933576 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":321,"threads_started":1}
I20260812 16:49:46.412751 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:46.411837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:48557)
0812 16:49:46.412438 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:46.412546 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":355,"threads_started":1}
I20260812 16:49:46.857548 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:46.856666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:49393)
0812 16:49:46.857264 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:46.857371 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
I20260812 16:49:46.936372 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:46.935401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:60015)
0812 16:49:46.936076 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:46.936174 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":366,"threads_started":1}
I20260812 16:49:47.415354 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:47.414264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:35749)
0812 16:49:47.415033 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:47.415158 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":385,"threads_started":1}
I20260812 16:49:47.859831 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:47.858713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:50927)
0812 16:49:47.859370 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:47.859490 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":343,"threads_started":1}
I20260812 16:49:47.938338 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:47.937476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:43107)
0812 16:49:47.938041 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:47.938160 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
I20260812 16:49:48.417348 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:48.416491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:47173)
0812 16:49:48.417030 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:48.417112 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":260,"threads_started":1}
I20260812 16:49:48.861650 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:48.860872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:39561)
0812 16:49:48.861368 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:48.861462 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":244,"threads_started":1}
I20260812 16:49:48.940528 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:48.939606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:56813)
0812 16:49:48.940194 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:48.940301 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260812 16:49:49.419222 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:49.418361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:51511)
0812 16:49:49.418908 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:49.419032 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":312,"threads_started":1}
I20260812 16:49:49.863521 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:49.862673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:55803)
0812 16:49:49.863241 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:49.863330 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":295,"threads_started":1}
I20260812 16:49:49.942555 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:49.941751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:34501)
0812 16:49:49.942286 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:49.942379 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":285,"threads_started":1}
I20260812 16:49:50.421023 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:50.420200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:59851)
0812 16:49:50.420737 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:50.420818 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":274,"threads_started":1}
I20260812 16:49:50.865837 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:50.864654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:49169)
0812 16:49:50.865477 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:50.865613 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":447,"threads_started":1}
I20260812 16:49:50.944378 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:50.943577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:58655)
0812 16:49:50.944097 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:50.944185 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":261,"threads_started":1}
I20260812 16:49:51.422995 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:51.422111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:41457)
0812 16:49:51.422660 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:51.422756 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":301,"threads_started":1}
I20260812 16:49:51.868330 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:51.867380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:53467)
0812 16:49:51.867969 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:51.868081 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":325,"threads_started":1}
I20260812 16:49:51.946856 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:51.945886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:33185)
0812 16:49:51.946552 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:51.946673 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":346,"threads_started":1}
I20260812 16:49:52.424942 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:52.424059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:51789)
0812 16:49:52.424618 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:52.424733 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":297,"threads_started":1}
I20260812 16:49:52.870843 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:52.869885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:55871)
0812 16:49:52.870538 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:52.870658 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":347,"threads_started":1}
I20260812 16:49:52.949612 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:52.948509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:40587)
0812 16:49:52.949270 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:52.949406 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":287,"threads_started":1}
I20260812 16:49:53.427908 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:53.426606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:51299)
0812 16:49:53.427568 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:53.427714 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":552,"threads_started":1}
I20260812 16:49:53.873026 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:53.872041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:41415)
0812 16:49:53.872696 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:53.872822 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":289,"threads_started":1}
I20260812 16:49:53.952353 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:53.950878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:34331)
0812 16:49:53.951656 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:53.951883 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":357,"threads_started":1}
I20260812 16:49:54.431304 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:54.430186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:60549)
0812 16:49:54.430992 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:54.431098 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":403,"threads_started":1}
I20260812 16:49:54.876257 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:54.874448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:58025)
0812 16:49:54.875254 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:54.875974 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":325,"threads_started":1}
I20260812 16:49:54.955971 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:54.954508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:37511)
0812 16:49:54.955562 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:54.955706 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":708,"threads_started":1}
I20260812 16:49:55.433918 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:55.432703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:42715)
0812 16:49:55.433411 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:55.433539 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":427,"threads_started":1}
I20260812 16:49:55.878398 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:55.877575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:46293)
0812 16:49:55.878093 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:55.878199 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260812 16:49:55.958026 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:55.957178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:46785)
0812 16:49:55.957727 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:55.957825 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":269,"threads_started":1}
I20260812 16:49:56.436816 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:56.435910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.2:49927)
0812 16:49:56.436464 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:56.436602 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260812 16:49:56.880363 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:56.879507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.1:55797)
0812 16:49:56.880079 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:56.880193 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
I20260812 16:49:56.923821 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 25939
I20260812 16:49:56.956566 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:56.925185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.28.243.1:35657)
0812 16:49:56.925904 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:56.925919 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:56.925969 (+ 50us) server_negotiation.cc:407] Connection header received
0812 16:49:56.928070 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:56.928083 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:56.928601 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:56.929260 (+ 659us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:56.931537 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:56.933220 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:56.935249 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:56.936164 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:56.936378 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:56.945017 (+ 8639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:56.945029 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:56.945052 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:56.945110 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:56.950569 (+ 5459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:56.951978 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:56.951987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:56.951996 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:56.952260 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:56.953031 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:56.953039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:56.953042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:56.954334 (+ 1292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:56.954778 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:56.955529 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 16:49:56.956162 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":351,"threads_started":1}
I20260812 16:49:56.958725 26231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:56.960404 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:56.959359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:38365 (local address 127.28.243.3:48993)
0812 16:49:56.959912 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 16:49:56.960150 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.62:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":276,"threads_started":1}
I20260812 16:49:57.052238 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 25939
I20260812 16:49:57.086475 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 26245
I20260812 16:49:57.110124 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:57.087735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.28.243.2:41195)
0812 16:49:57.089097 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:57.089118 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:57.089177 (+ 59us) server_negotiation.cc:407] Connection header received
0812 16:49:57.091467 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:57.091484 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:57.092133 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:57.092536 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:57.094613 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:57.096574 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:57.098122 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:57.099114 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:57.099336 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:57.102965 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:57.102982 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:57.103007 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:57.103079 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:57.106260 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:57.107249 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:57.107258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:57.107266 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:57.107478 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:57.107825 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:57.107828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:57.107830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:57.108461 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:57.108726 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:57.109204 (+ 478us) server_negotiation.cc:299] Negotiation successful
0812 16:49:57.109799 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":781,"threads_started":1}
I20260812 16:49:57.114500 26473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:57.211978 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 26245
I20260812 16:49:57.248672 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 26482
I20260812 16:49:57.273536 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:57.249848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.28.243.3:40495)
0812 16:49:57.250778 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:57.250799 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:57.251007 (+ 208us) server_negotiation.cc:407] Connection header received
0812 16:49:57.252854 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:57.252867 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:57.253718 (+ 851us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:57.254333 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:57.256772 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:57.258168 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:57.259656 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:57.260450 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:57.260621 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:57.263575 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:57.263587 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:57.263608 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:57.263671 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:57.267331 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:57.269018 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:57.269051 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:57.269065 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:57.269426 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:57.270227 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:57.270235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:57.270237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:57.271405 (+ 1168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:57.271774 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:57.272407 (+ 633us) server_negotiation.cc:299] Negotiation successful
0812 16:49:57.273117 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":429,"threads_started":1}
I20260812 16:49:57.275905 26708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:49:57.369663 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 26482
2026-08-12T16:49:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66750 ms)
[----------] 30 tests from SecurityITest (341142 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:49:57 dist-test-slave-1vmg krb5kdc[26813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:49:57 dist-test-slave-1vmg krb5kdc[26813](info): set up 2 sockets
Aug 12 16:49:57 dist-test-slave-1vmg krb5kdc[26813](info): commencing operation
krb5kdc: starting...
W20260812 16:49:58.760749 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:49:58 dist-test-slave-1vmg krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:49:59.131613 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42041
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:49:59.522285 26835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:49:59.522647 26835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:49:59.522791 26835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:49:59.530844 26835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:49:59.531042 26835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:49:59.531141 26835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:49:59.531224 26835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:49:59.531288 26835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:49:59.542449 26835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:42041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:49:59.543974 26835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:49:59.545387 26835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:49:59.559095 26843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:59.559111 26844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:49:59.559183 26848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:49:59.560053 26835 server_base.cc:1034] running on GCE node
I20260812 16:49:59.560935 26835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:49:59.562570 26835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:49:59.563859 26835 hybrid_clock.cc:648] HybridClock initialized: now 1786553399563801 us; error 42 us; skew 500 ppm
I20260812 16:49:59.564410 26835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:49:59 dist-test-slave-1vmg krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553399, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:49:59.569073 26835 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:49:59.571681 26835 webserver.cc:460] Webserver started at http://127.28.243.62:36003/ using document root <none> and password file <none>
I20260812 16:49:59.572708 26835 fs_manager.cc:359] Metadata directory not provided
I20260812 16:49:59.572862 26835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:49:59.573262 26835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:49:59.577639 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "2151664c8d664378b88ce58013557e48"
format_stamp: "Formatted at 2026-08-12 16:49:59 on dist-test-slave-1vmg"
server_key: "e21b4fdf7cca8767f1c15bc51cbeb00d"
server_key_iv: "b0f3020f91e25f4ee7529c63e463a9fe"
server_key_version: "encryptionkey@0"
I20260812 16:49:59.578615 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "2151664c8d664378b88ce58013557e48"
format_stamp: "Formatted at 2026-08-12 16:49:59 on dist-test-slave-1vmg"
server_key: "e21b4fdf7cca8767f1c15bc51cbeb00d"
server_key_iv: "b0f3020f91e25f4ee7529c63e463a9fe"
server_key_version: "encryptionkey@0"
I20260812 16:49:59.585024 26835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 16:49:59.589741 26859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:49:59.592336 26835 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 16:49:59.592662 26835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "2151664c8d664378b88ce58013557e48"
format_stamp: "Formatted at 2026-08-12 16:49:59 on dist-test-slave-1vmg"
server_key: "e21b4fdf7cca8767f1c15bc51cbeb00d"
server_key_iv: "b0f3020f91e25f4ee7529c63e463a9fe"
server_key_version: "encryptionkey@0"
I20260812 16:49:59.593813 26835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:49:59.621209 26835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:49:59.626783 26835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:49:59.627182 26835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:49:59.650550 26835 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:42041
I20260812 16:49:59.650570 26947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:42041 every 8 connection(s)
I20260812 16:49:59.652052 26835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:49:59.656487 26952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:49:59.658589 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 26835
I20260812 16:49:59.658841 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:49:59.659775 29644 external_mini_cluster.cc:1428] Setting key c83165f556e0ad4ddbeb71ef36949a27
I20260812 16:49:59.670529 26952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48: Bootstrap starting.
I20260812 16:49:59.674873 26952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48: Neither blocks nor log segments found. Creating new log.
I20260812 16:49:59.676455 26952 log.cc:824] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:49:59 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:49:59.680374 26952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48: No bootstrap required, opened a new log
I20260812 16:49:59.689474 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:49:59.662227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.28.243.62:42041)
0812 16:49:59.663191 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0812 16:49:59.663208 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:49:59.663261 (+ 53us) server_negotiation.cc:407] Connection header received
0812 16:49:59.665112 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:49:59.665141 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:49:59.665614 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:49:59.666256 (+ 642us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:49:59.668361 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:59.670426 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:59.672738 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:49:59.673667 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:49:59.673848 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:49:59.679804 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:49:59.679813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:49:59.679830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:49:59.679878 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:49:59.683336 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:59.684395 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:59.684403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:59.684410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:59.684622 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:49:59.685276 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:49:59.685281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:49:59.685283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:49:59.686212 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:49:59.686706 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0812 16:49:59.687436 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 16:49:59.687945 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":389,"threads_started":1}
I20260812 16:49:59.690140 26952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2151664c8d664378b88ce58013557e48" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42041 } }
I20260812 16:49:59.690665 26952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:49:59.690779 26952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2151664c8d664378b88ce58013557e48, State: Initialized, Role: FOLLOWER
I20260812 16:49:59.691495 26952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2151664c8d664378b88ce58013557e48" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42041 } }
I20260812 16:49:59.691787 26952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:49:59.692041 26952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:49:59.692200 26952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:49:59.695181 26952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2151664c8d664378b88ce58013557e48" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42041 } }
I20260812 16:49:59.695782 26952 leader_election.cc:302] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2151664c8d664378b88ce58013557e48; no voters:
I20260812 16:49:59.696611 26952 leader_election.cc:288] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:49:59.696770 26959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:49:59.697466 26959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [term 1 LEADER]: Becoming Leader. State: Replica: 2151664c8d664378b88ce58013557e48, State: Running, Role: LEADER
I20260812 16:49:59.698120 26959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2151664c8d664378b88ce58013557e48" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42041 } }
I20260812 16:49:59.699074 26952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:49:59.701550 26960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2151664c8d664378b88ce58013557e48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2151664c8d664378b88ce58013557e48" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42041 } } }
I20260812 16:49:59.702050 26960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [sys.catalog]: This master's current role is: LEADER
I20260812 16:49:59.702186 26961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2151664c8d664378b88ce58013557e48. Latest consensus state: current_term: 1 leader_uuid: "2151664c8d664378b88ce58013557e48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2151664c8d664378b88ce58013557e48" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 42041 } } }
I20260812 16:49:59.702540 26961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48 [sys.catalog]: This master's current role is: LEADER
I20260812 16:49:59.708981 26970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:49:59.718828 26970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:49:59.729578 26970 catalog_manager.cc:1347] Generated new cluster ID: 64fc9f7aaa40403aa0fff63a0a7c672d
I20260812 16:49:59.729775 26970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:49:59.749177 26970 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:49:59.751959 26970 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:49:59.762775 26970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48: Generated new TSK 0
I20260812 16:49:59.763823 26970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:49:59.878090 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42041
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:00.295220 27000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:00.295691 27000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:00.295856 27000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:00.305007 27000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:00.305181 27000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:00.305366 27000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:50:00.317806 27000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:00.319358 27000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:00.320885 27000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:00.337399 27012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:00.337368 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:00.337324 27014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:00.338248 27000 server_base.cc:1034] running on GCE node
I20260812 16:50:00.338973 27000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:00.340194 27000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:00.341689 27000 hybrid_clock.cc:648] HybridClock initialized: now 1786553400341612 us; error 88 us; skew 500 ppm
I20260812 16:50:00.342132 27000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:00 dist-test-slave-1vmg krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553400, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:00.345988 27000 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:50:00.348793 27000 webserver.cc:460] Webserver started at http://127.28.243.1:46345/ using document root <none> and password file <none>
I20260812 16:50:00.349836 27000 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:00.349980 27000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:00.350361 27000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:00.355249 27000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "c431b6a8571f4427b8660febb8ebc5b1"
format_stamp: "Formatted at 2026-08-12 16:50:00 on dist-test-slave-1vmg"
server_key: "0cd1d8dcc21c5e2e7a0b25b9b236d0a3"
server_key_iv: "e351353cdf63ca885fb9b656e30ab60d"
server_key_version: "encryptionkey@0"
I20260812 16:50:00.356506 27000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "c431b6a8571f4427b8660febb8ebc5b1"
format_stamp: "Formatted at 2026-08-12 16:50:00 on dist-test-slave-1vmg"
server_key: "0cd1d8dcc21c5e2e7a0b25b9b236d0a3"
server_key_iv: "e351353cdf63ca885fb9b656e30ab60d"
server_key_version: "encryptionkey@0"
I20260812 16:50:00.363689 27000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:50:00.368940 27030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:00.371634 27000 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:50:00.371878 27000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "c431b6a8571f4427b8660febb8ebc5b1"
format_stamp: "Formatted at 2026-08-12 16:50:00 on dist-test-slave-1vmg"
server_key: "0cd1d8dcc21c5e2e7a0b25b9b236d0a3"
server_key_iv: "e351353cdf63ca885fb9b656e30ab60d"
server_key_version: "encryptionkey@0"
I20260812 16:50:00.372449 27000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:00.392381 27000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:00.398213 27000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:00.398547 27000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:00.399834 27000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:00.401425 27000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:00.401515 27000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:00.401665 27000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:00.401726 27000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:00.435683 27000 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:33265
I20260812 16:50:00.435756 27220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:33265 every 8 connection(s)
I20260812 16:50:00.437058 27000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:50:00.440716 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 27000
I20260812 16:50:00.440954 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:50:00.441896 29644 external_mini_cluster.cc:1428] Setting key 26fbf2f6e836740450210f93981cfa89
Aug 12 16:50:00 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553400, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:00.461633 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:00.439688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:35535 (local address 127.28.243.62:42041)
0812 16:50:00.440169 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:00.440174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:00.441237 (+ 1063us) server_negotiation.cc:407] Connection header received
0812 16:50:00.443798 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:00.443805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:00.443896 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:00.444003 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:00.445724 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:00.446741 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:00.449478 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:00.450266 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:00.450568 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:00.454435 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:00.454443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:00.454447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:00.454493 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:00.457827 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:00.458782 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:00.458788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:00.458791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:00.458999 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:00.459599 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:00.459602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:00.459604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:00.459923 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:00.460050 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:00.461117 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0812 16:50:00.461334 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
I20260812 16:50:00.463050 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:00.440149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42041 (local address 127.28.243.1:35535)
0812 16:50:00.441025 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:00.441095 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:00.443470 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:00.444296 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:00.444306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:00.444875 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:00.445512 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:00.445532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:00.446903 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:00.446907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:00.449156 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:00.449191 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:00.450627 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:00.450634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:00.450894 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:00.451738 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:00.451764 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:00.454105 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:00.458053 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:00.458063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:00.458081 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:00.458618 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:00.459195 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:00.459199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:00.459202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:00.459448 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:00.460061 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:00.460071 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:00.460599 (+ 528us) client_negotiation.cc:769] Sending connection context
0812 16:50:00.460953 (+ 354us) client_negotiation.cc:240] Negotiation successful
0812 16:50:00.461570 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":327,"threads_started":1}
I20260812 16:50:00.464818 27222 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42041
I20260812 16:50:00.465193 27222 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:00.466158 27222 heartbeater.cc:511] Master 127.28.243.62:42041 requested a full tablet report, sending...
I20260812 16:50:00.469743 26884 ts_manager.cc:194] Registered new tserver with Master: c431b6a8571f4427b8660febb8ebc5b1 (127.28.243.1:33265)
I20260812 16:50:00.472106 26884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:35535
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:50:00.522818 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42041
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:00.930658 27238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:00.931051 27238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:00.931221 27238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:00.938855 27238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:00.939173 27238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:00.939361 27238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:50:00.950071 27238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:00.951601 27238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:00.953038 27238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:00.968001 27248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:00.967948 27257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:00.967914 27246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:00.968838 27238 server_base.cc:1034] running on GCE node
I20260812 16:50:00.969503 27238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:00.970522 27238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:00.971737 27238 hybrid_clock.cc:648] HybridClock initialized: now 1786553400971674 us; error 53 us; skew 500 ppm
I20260812 16:50:00.972090 27238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:00 dist-test-slave-1vmg krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553400, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:00.975363 27238 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:50:00.976979 27238 webserver.cc:460] Webserver started at http://127.28.243.2:45011/ using document root <none> and password file <none>
I20260812 16:50:00.977780 27238 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:00.977912 27238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:00.978242 27238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:00.981400 27238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "d46ac544ce70420da317cafa90fe86aa"
format_stamp: "Formatted at 2026-08-12 16:50:00 on dist-test-slave-1vmg"
server_key: "e1edfd354bd7607eed798dae88e82580"
server_key_iv: "5a89c514e765048e8244657b4214e4a8"
server_key_version: "encryptionkey@0"
I20260812 16:50:00.982195 27238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "d46ac544ce70420da317cafa90fe86aa"
format_stamp: "Formatted at 2026-08-12 16:50:00 on dist-test-slave-1vmg"
server_key: "e1edfd354bd7607eed798dae88e82580"
server_key_iv: "5a89c514e765048e8244657b4214e4a8"
server_key_version: "encryptionkey@0"
I20260812 16:50:00.987615 27238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:50:00.992168 27269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:00.993876 27238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:50:00.994089 27238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "d46ac544ce70420da317cafa90fe86aa"
format_stamp: "Formatted at 2026-08-12 16:50:00 on dist-test-slave-1vmg"
server_key: "e1edfd354bd7607eed798dae88e82580"
server_key_iv: "5a89c514e765048e8244657b4214e4a8"
server_key_version: "encryptionkey@0"
I20260812 16:50:00.994330 27238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:01.030536 27238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:01.035411 27238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:01.035723 27238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:01.036857 27238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:01.039404 27238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:01.039518 27238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:01.039672 27238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:01.039731 27238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:01.083104 27238 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:40383
I20260812 16:50:01.083149 27461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:40383 every 8 connection(s)
I20260812 16:50:01.084506 27238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:50:01.095706 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 27238
I20260812 16:50:01.095934 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:50:01.096562 29644 external_mini_cluster.cc:1428] Setting key cbc7d71f61fd4a54c753a784a2c20faa
Aug 12 16:50:01 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553400, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:01.110896 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.087848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:49359 (local address 127.28.243.62:42041)
0812 16:50:01.088370 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.088374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.090397 (+ 2023us) server_negotiation.cc:407] Connection header received
0812 16:50:01.092643 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.092650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.092746 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.092849 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:01.094677 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.095696 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.098371 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.099043 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.099168 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.104002 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:01.104008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:01.104011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:01.104050 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:01.106881 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:01.108367 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:01.108372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:01.108374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:01.108531 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:01.109069 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:01.109072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:01.109074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:01.109385 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:01.109501 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.110414 (+ 913us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.110610 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":293,"threads_started":1}
I20260812 16:50:01.112422 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.088621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42041 (local address 127.28.243.2:49359)
0812 16:50:01.090118 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.090189 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.092270 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.093160 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.093170 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.093795 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:01.094446 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.094468 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.096046 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.096053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.098176 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.098194 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.099358 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.099368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.099848 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.100648 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:01.100690 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:01.103700 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:01.107143 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:01.107155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:01.107176 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:01.108196 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:01.108661 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:01.108666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:01.108669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:01.108948 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:01.109525 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:01.109534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:01.110013 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.110338 (+ 325us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.110963 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":690,"threads_started":1}
I20260812 16:50:01.114209 27463 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42041
I20260812 16:50:01.114526 27463 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:01.115365 27463 heartbeater.cc:511] Master 127.28.243.62:42041 requested a full tablet report, sending...
I20260812 16:50:01.117084 26884 ts_manager.cc:194] Registered new tserver with Master: d46ac544ce70420da317cafa90fe86aa (127.28.243.2:40383)
I20260812 16:50:01.118268 26884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:49359
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:50:01.178867 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42041
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:50:01.478006 27222 heartbeater.cc:503] Master 127.28.243.62:42041 was elected leader, sending a full tablet report...
W20260812 16:50:01.631929 27472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:01.632239 27472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:01.632369 27472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:01.639904 27472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:01.640038 27472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:01.640195 27472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:50:01.651837 27472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:01.654258 27472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:01.656876 27472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:01.673022 27479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:01.673076 27484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:01.673247 27478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:01.674012 27472 server_base.cc:1034] running on GCE node
I20260812 16:50:01.675104 27472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:01.676932 27472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:01.678261 27472 hybrid_clock.cc:648] HybridClock initialized: now 1786553401678218 us; error 72 us; skew 500 ppm
I20260812 16:50:01.678710 27472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:01 dist-test-slave-1vmg krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553401, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:01.682528 27472 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:50:01.684414 27472 webserver.cc:460] Webserver started at http://127.28.243.3:36469/ using document root <none> and password file <none>
I20260812 16:50:01.685209 27472 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:01.685356 27472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:01.685714 27472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:01.689740 27472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "d19314bc5d864461af1b4ed17f430cd8"
format_stamp: "Formatted at 2026-08-12 16:50:01 on dist-test-slave-1vmg"
server_key: "8f8b540e0399a16e02b7c0bb5525685b"
server_key_iv: "befce23e1f97b4b98e9b11cf25b9b1ab"
server_key_version: "encryptionkey@0"
I20260812 16:50:01.690659 27472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "d19314bc5d864461af1b4ed17f430cd8"
format_stamp: "Formatted at 2026-08-12 16:50:01 on dist-test-slave-1vmg"
server_key: "8f8b540e0399a16e02b7c0bb5525685b"
server_key_iv: "befce23e1f97b4b98e9b11cf25b9b1ab"
server_key_version: "encryptionkey@0"
I20260812 16:50:01.698801 27472 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 16:50:01.703842 27495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:01.705686 27472 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:50:01.705893 27472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "d19314bc5d864461af1b4ed17f430cd8"
format_stamp: "Formatted at 2026-08-12 16:50:01 on dist-test-slave-1vmg"
server_key: "8f8b540e0399a16e02b7c0bb5525685b"
server_key_iv: "befce23e1f97b4b98e9b11cf25b9b1ab"
server_key_version: "encryptionkey@0"
I20260812 16:50:01.706117 27472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:01.744431 27472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:01.751135 27472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:01.751556 27472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:01.753247 27472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:01.755329 27472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:01.755573 27472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:01.755848 27472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:01.755918 27472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:01.797068 27472 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:45063
I20260812 16:50:01.797235 27705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:45063 every 8 connection(s)
I20260812 16:50:01.798650 27472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:50:01.801798 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 27472
I20260812 16:50:01.802008 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:50:01.802637 29644 external_mini_cluster.cc:1428] Setting key a5a17e2429b38b44289dea917f0f4271
Aug 12 16:50:01 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553401, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:01.822154 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.801786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:50137 (local address 127.28.243.62:42041)
0812 16:50:01.804272 (+ 2486us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.804278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.804302 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:50:01.805741 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.805746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.805816 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.805915 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:01.807652 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.808455 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.810060 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.810487 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.810582 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.814595 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:01.814603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:01.814607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:01.814691 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:01.818342 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:01.819567 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:01.819574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:01.819577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:01.819742 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:01.820268 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:01.820273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:01.820275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:01.820588 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:01.820761 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.821613 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.821833 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2379,"thread_start_us":2231,"threads_started":1}
I20260812 16:50:01.823237 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.802375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:42041 (local address 127.28.243.3:50137)
0812 16:50:01.803330 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.803383 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.805446 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.806225 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.806238 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.806804 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:01.807477 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.807495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.808595 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.808600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.809869 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.809884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.810727 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.810733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.811047 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.811792 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:01.811823 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:01.814191 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:01.818530 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:01.818541 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:01.818564 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:01.819425 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:01.819864 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:01.819868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:01.819870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:01.820085 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:01.820746 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:01.820754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:01.821167 (+ 413us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.821433 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.821908 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":441,"threads_started":1}
I20260812 16:50:01.825335 27706 heartbeater.cc:347] Connected to a master server at 127.28.243.62:42041
I20260812 16:50:01.825718 27706 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:01.826692 27706 heartbeater.cc:511] Master 127.28.243.62:42041 requested a full tablet report, sending...
I20260812 16:50:01.828594 26884 ts_manager.cc:194] Registered new tserver with Master: d19314bc5d864461af1b4ed17f430cd8 (127.28.243.3:45063)
I20260812 16:50:01.829597 26884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:50137
I20260812 16:50:01.833559 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:50:01.856703 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.839812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.28.243.62:42041)
0812 16:50:01.840044 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.840051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.840474 (+ 423us) server_negotiation.cc:407] Connection header received
0812 16:50:01.840670 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.840675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.840772 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.840862 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:01.842500 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.843328 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.844805 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.845231 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.845441 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.848853 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:01.848861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:01.848864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:01.848929 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:01.852432 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:01.853601 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:01.853609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:01.853613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:01.853867 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:01.854487 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:01.854492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:01.854494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:01.855030 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:01.855445 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.856044 (+ 599us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.856352 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:50:01.862349 26884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:50:01.869644 26884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:50:01.904865 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.887309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:45063 (local address 127.0.0.1:59936)
0812 16:50:01.888611 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.888668 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.888897 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.893689 (+ 4792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.893700 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.893746 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:01.895491 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.895513 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.898333 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.898340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.901263 (+ 2923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.901292 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.903769 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.903776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.904007 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.904077 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.904201 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.904451 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":559,"threads_started":1}
I20260812 16:50:01.906086 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.887854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.28.243.3:45063)
0812 16:50:01.892059 (+ 4205us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.892104 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.892367 (+ 263us) server_negotiation.cc:407] Connection header received
0812 16:50:01.892554 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.892565 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.893153 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.895851 (+ 2698us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:01.896261 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.898098 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.901664 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.903515 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.905236 (+ 1721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.905541 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.905640 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.905817 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3737,"thread_start_us":3541,"threads_started":1}
I20260812 16:50:01.910264 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.889067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.28.243.1:33265)
0812 16:50:01.895309 (+ 6242us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.895321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.895359 (+ 38us) server_negotiation.cc:407] Connection header received
0812 16:50:01.895490 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.895500 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.895847 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.896443 (+ 596us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:01.897615 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.901393 (+ 3778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.905711 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.908415 (+ 2704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.908757 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.909091 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.909506 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.909742 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6026,"thread_start_us":5580,"threads_started":1}
I20260812 16:50:01.912037 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.888346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:33265 (local address 127.0.0.1:52312)
0812 16:50:01.892555 (+ 4209us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.892587 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.892773 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.896810 (+ 4037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.896816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.896852 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:01.897362 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.897375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.901624 (+ 4249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.901631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.905503 (+ 3872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.905522 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.908909 (+ 3387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.908919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.909145 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.909193 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.909460 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.909675 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3899,"thread_start_us":3575,"threads_started":1}
I20260812 16:50:01.911792 27587 tablet_service.cc:1467] Processing CreateTablet for tablet f27d4d43f75f470480cda6b00171bd33 (DEFAULT_TABLE table=test-table [id=16936aca82b64adfb4b655e9e1cdf25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:01.913795 27587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27d4d43f75f470480cda6b00171bd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:01.917385 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.895692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:40383 (local address 127.0.0.1:45618)
0812 16:50:01.903327 (+ 7635us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.903354 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.903544 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.905453 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.905459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.905495 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:01.905877 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.905891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.909537 (+ 3646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.909544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.913237 (+ 3693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.913265 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.916323 (+ 3058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.916332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.916553 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.916600 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.916741 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.916916 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7444,"thread_start_us":7941,"threads_started":1}
I20260812 16:50:01.917397 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.893969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.28.243.2:40383)
0812 16:50:01.904358 (+ 10389us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.904367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.904398 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:50:01.904516 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.904525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.904841 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.907083 (+ 2242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:01.907502 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.909317 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.913511 (+ 4194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.915959 (+ 2448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.916195 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.916428 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.916830 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.917043 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10199,"thread_start_us":10043,"threads_started":1}
I20260812 16:50:01.918107 27116 tablet_service.cc:1467] Processing CreateTablet for tablet f27d4d43f75f470480cda6b00171bd33 (DEFAULT_TABLE table=test-table [id=16936aca82b64adfb4b655e9e1cdf25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:01.919821 27116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27d4d43f75f470480cda6b00171bd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:01.922231 27348 tablet_service.cc:1467] Processing CreateTablet for tablet f27d4d43f75f470480cda6b00171bd33 (DEFAULT_TABLE table=test-table [id=16936aca82b64adfb4b655e9e1cdf25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:01.923954 27348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27d4d43f75f470480cda6b00171bd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:01.927533 27735 tablet_bootstrap.cc:492] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Bootstrap starting.
I20260812 16:50:01.932276 27735 tablet_bootstrap.cc:654] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:01.933431 27735 log.cc:824] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:01.936053 27740 tablet_bootstrap.cc:492] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Bootstrap starting.
I20260812 16:50:01.937031 27735 tablet_bootstrap.cc:492] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: No bootstrap required, opened a new log
I20260812 16:50:01.937487 27735 ts_tablet_manager.cc:1397] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260812 16:50:01.940533 27742 tablet_bootstrap.cc:492] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Bootstrap starting.
I20260812 16:50:01.940639 27740 tablet_bootstrap.cc:654] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:01.942142 27740 log.cc:824] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:01.943960 27742 tablet_bootstrap.cc:654] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:01.945042 27742 log.cc:824] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:01.946552 27740 tablet_bootstrap.cc:492] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: No bootstrap required, opened a new log
I20260812 16:50:01.946952 27740 ts_tablet_manager.cc:1397] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 16:50:01.948102 27742 tablet_bootstrap.cc:492] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: No bootstrap required, opened a new log
I20260812 16:50:01.948573 27742 ts_tablet_manager.cc:1397] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 16:50:01.950294 27735 raft_consensus.cc:348] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.950881 27735 raft_consensus.cc:374] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:01.951021 27735 raft_consensus.cc:729] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19314bc5d864461af1b4ed17f430cd8, State: Initialized, Role: FOLLOWER
I20260812 16:50:01.951792 27735 consensus_queue.cc:260] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.956245 27706 heartbeater.cc:503] Master 127.28.243.62:42041 was elected leader, sending a full tablet report...
I20260812 16:50:01.957370 27742 raft_consensus.cc:348] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.957824 27742 raft_consensus.cc:374] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:01.957983 27742 raft_consensus.cc:729] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46ac544ce70420da317cafa90fe86aa, State: Initialized, Role: FOLLOWER
I20260812 16:50:01.959100 27735 ts_tablet_manager.cc:1428] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Time spent starting tablet: real 0.021s user 0.016s sys 0.002s
I20260812 16:50:01.959254 27742 consensus_queue.cc:260] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.959302 27740 raft_consensus.cc:348] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.959813 27740 raft_consensus.cc:374] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:01.959936 27740 raft_consensus.cc:729] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c431b6a8571f4427b8660febb8ebc5b1, State: Initialized, Role: FOLLOWER
I20260812 16:50:01.960602 27740 consensus_queue.cc:260] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.964077 27740 ts_tablet_manager.cc:1428] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260812 16:50:01.964208 27742 ts_tablet_manager.cc:1428] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260812 16:50:01.964687 27463 heartbeater.cc:503] Master 127.28.243.62:42041 was elected leader, sending a full tablet report...
I20260812 16:50:01.984133 27750 raft_consensus.cc:482] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:01.984452 27750 raft_consensus.cc:504] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:01.986300 27750 leader_election.cc:288] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19314bc5d864461af1b4ed17f430cd8 (127.28.243.3:45063), c431b6a8571f4427b8660febb8ebc5b1 (127.28.243.1:33265)
I20260812 16:50:01.997789 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.986880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:35335 (local address 127.28.243.3:45063)
0812 16:50:01.987101 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.987108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.987925 (+ 817us) server_negotiation.cc:407] Connection header received
0812 16:50:01.988159 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.988170 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.988270 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.988518 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:01.989567 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.990727 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.994178 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.995875 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.996102 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.996309 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.997196 (+ 887us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.997510 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:50:01.997789 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.987115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:39847 (local address 127.28.243.1:33265)
0812 16:50:01.987365 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:01.987372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:01.987652 (+ 280us) server_negotiation.cc:407] Connection header received
0812 16:50:01.987960 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:01.987967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:01.988230 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:01.988518 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:01.989590 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.990727 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.994195 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.995874 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.996203 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.996308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:01.997196 (+ 888us) server_negotiation.cc:299] Negotiation successful
0812 16:50:01.997510 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:50:01.997980 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.986746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:45063 (local address 127.28.243.2:35335)
0812 16:50:01.987793 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.987809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.987923 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.988537 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.988543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.988576 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:01.989256 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.989298 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.990934 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.990941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.993899 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.993924 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.996265 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.996271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.996726 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.996829 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.997140 (+ 311us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.997414 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":294,"thread_start_us":661,"threads_started":1}
I20260812 16:50:01.997980 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:01.986703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:33265 (local address 127.28.243.2:39847)
0812 16:50:01.987505 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:01.987530 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:01.987742 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:01.988729 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:01.988733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:01.988828 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:01.989396 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.989411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.990936 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.990941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.993900 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:01.993924 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:01.996263 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:01.996371 (+ 108us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:01.996721 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:01.996829 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 16:50:01.997141 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 16:50:01.997414 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":422,"spinlock_wait_cycles":29056,"thread_start_us":435,"threads_started":1}
I20260812 16:50:01.999362 27150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27d4d43f75f470480cda6b00171bd33" candidate_uuid: "d46ac544ce70420da317cafa90fe86aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c431b6a8571f4427b8660febb8ebc5b1" is_pre_election: true
I20260812 16:50:01.999629 27622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27d4d43f75f470480cda6b00171bd33" candidate_uuid: "d46ac544ce70420da317cafa90fe86aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19314bc5d864461af1b4ed17f430cd8" is_pre_election: true
I20260812 16:50:02.000001 27150 raft_consensus.cc:2454] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d46ac544ce70420da317cafa90fe86aa in term 0.
I20260812 16:50:02.000568 27622 raft_consensus.cc:2454] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d46ac544ce70420da317cafa90fe86aa in term 0.
I20260812 16:50:02.001178 27271 leader_election.cc:302] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c431b6a8571f4427b8660febb8ebc5b1, d46ac544ce70420da317cafa90fe86aa; no voters:
I20260812 16:50:02.001803 27750 raft_consensus.cc:2781] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:02.001976 27750 raft_consensus.cc:482] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:02.002074 27750 raft_consensus.cc:3037] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:02.006400 27750 raft_consensus.cc:504] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:02.007273 27750 leader_election.cc:288] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [CANDIDATE]: Term 1 election: Requested vote from peers d19314bc5d864461af1b4ed17f430cd8 (127.28.243.3:45063), c431b6a8571f4427b8660febb8ebc5b1 (127.28.243.1:33265)
I20260812 16:50:02.007951 27622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27d4d43f75f470480cda6b00171bd33" candidate_uuid: "d46ac544ce70420da317cafa90fe86aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19314bc5d864461af1b4ed17f430cd8"
I20260812 16:50:02.008001 27150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27d4d43f75f470480cda6b00171bd33" candidate_uuid: "d46ac544ce70420da317cafa90fe86aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c431b6a8571f4427b8660febb8ebc5b1"
I20260812 16:50:02.008194 27622 raft_consensus.cc:3037] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:02.008333 27150 raft_consensus.cc:3037] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:02.010742 27150 raft_consensus.cc:2454] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46ac544ce70420da317cafa90fe86aa in term 1.
I20260812 16:50:02.010771 27622 raft_consensus.cc:2454] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46ac544ce70420da317cafa90fe86aa in term 1.
I20260812 16:50:02.011334 27271 leader_election.cc:302] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c431b6a8571f4427b8660febb8ebc5b1, d19314bc5d864461af1b4ed17f430cd8, d46ac544ce70420da317cafa90fe86aa; no voters:
I20260812 16:50:02.011752 27750 raft_consensus.cc:2781] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:02.012311 27750 raft_consensus.cc:686] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 1 LEADER]: Becoming Leader. State: Replica: d46ac544ce70420da317cafa90fe86aa, State: Running, Role: LEADER
I20260812 16:50:02.012912 27750 consensus_queue.cc:237] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } }
I20260812 16:50:02.019485 26884 catalog_manager.cc:5495] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa reported cstate change: term changed from 0 to 1, leader changed from <none> to d46ac544ce70420da317cafa90fe86aa (127.28.243.2). New cstate: current_term: 1 leader_uuid: "d46ac544ce70420da317cafa90fe86aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d46ac544ce70420da317cafa90fe86aa" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40383 } health_report { overall_health: HEALTHY } } }
W20260812 16:50:02.050593 27707 tablet.cc:2369] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:02.061295 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:02.054249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.28.243.2:40383)
0812 16:50:02.054580 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:02.054585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:02.054605 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:50:02.054718 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:02.054723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:02.054821 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:02.055003 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:50:02.055623 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.056443 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.058219 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.058675 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.058792 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:02.059765 (+ 973us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:50:02.060360 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:50:02.060494 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:02.060834 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 16:50:02.061019 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260812 16:50:02.071934 27622 raft_consensus.cc:1261] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 1 FOLLOWER]: Refusing update from remote peer d46ac544ce70420da317cafa90fe86aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:02.071934 27150 raft_consensus.cc:1261] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 1 FOLLOWER]: Refusing update from remote peer d46ac544ce70420da317cafa90fe86aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:02.073351 27755 consensus_queue.cc:1035] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c431b6a8571f4427b8660febb8ebc5b1" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:02.073876 27750 consensus_queue.cc:1035] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19314bc5d864461af1b4ed17f430cd8" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:02.109472 26884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:02.109817 26884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:02.114991 26884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2151664c8d664378b88ce58013557e48: Sending DeleteTablet for 3 replicas of tablet f27d4d43f75f470480cda6b00171bd33
I20260812 16:50:02.116492 27587 tablet_service.cc:1514] Processing DeleteTablet for tablet f27d4d43f75f470480cda6b00171bd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:59936
I20260812 16:50:02.116595 27584 tablet_service.cc:1514] Processing DeleteTablet for tablet f27d4d43f75f470480cda6b00171bd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:59936
I20260812 16:50:02.118451 27116 tablet_service.cc:1514] Processing DeleteTablet for tablet f27d4d43f75f470480cda6b00171bd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:52312
I20260812 16:50:02.120478 27348 tablet_service.cc:1514] Processing DeleteTablet for tablet f27d4d43f75f470480cda6b00171bd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:02 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:45618
I20260812 16:50:02.122697 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 27000
I20260812 16:50:02.133912 27780 tablet_replica.cc:330] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: stopping tablet replica
I20260812 16:50:02.134485 27780 raft_consensus.cc:2229] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:02.134837 27781 tablet_replica.cc:330] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: stopping tablet replica
I20260812 16:50:02.135141 27780 raft_consensus.cc:2258] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:02.140009 27781 raft_consensus.cc:2229] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:02.140877 27781 raft_consensus.cc:2258] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:02.143576 27780 ts_tablet_manager.cc:1905] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:02.149642 27781 ts_tablet_manager.cc:1905] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:02.151253 27780 ts_tablet_manager.cc:1918] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:02.151412 27782 tablet_replica.cc:330] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: stopping tablet replica
I20260812 16:50:02.151470 27780 log.cc:1196] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/f27d4d43f75f470480cda6b00171bd33
I20260812 16:50:02.152050 27780 ts_tablet_manager.cc:1939] T f27d4d43f75f470480cda6b00171bd33 P d19314bc5d864461af1b4ed17f430cd8: Deleting consensus metadata
I20260812 16:50:02.152062 27782 raft_consensus.cc:2229] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:50:02.153064 27782 raft_consensus.cc:2258] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:02.155354 26865 catalog_manager.cc:4841] TS d19314bc5d864461af1b4ed17f430cd8 (127.28.243.3:45063): tablet f27d4d43f75f470480cda6b00171bd33 (table test-table [id=16936aca82b64adfb4b655e9e1cdf25f]) successfully deleted
W20260812 16:50:02.156141 26865 catalog_manager.cc:4808] TS d19314bc5d864461af1b4ed17f430cd8 (127.28.243.3:45063): delete failed for tablet f27d4d43f75f470480cda6b00171bd33 because the tablet was not found. No further retry: Not found: Tablet not found: f27d4d43f75f470480cda6b00171bd33
I20260812 16:50:02.158425 27782 ts_tablet_manager.cc:1905] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:02.158744 27781 ts_tablet_manager.cc:1918] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:02.159056 27781 log.cc:1196] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/f27d4d43f75f470480cda6b00171bd33
I20260812 16:50:02.159699 27781 ts_tablet_manager.cc:1939] T f27d4d43f75f470480cda6b00171bd33 P c431b6a8571f4427b8660febb8ebc5b1: Deleting consensus metadata
I20260812 16:50:02.161490 26861 catalog_manager.cc:4841] TS c431b6a8571f4427b8660febb8ebc5b1 (127.28.243.1:33265): tablet f27d4d43f75f470480cda6b00171bd33 (table test-table [id=16936aca82b64adfb4b655e9e1cdf25f]) successfully deleted
Aug 12 16:50:02 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:50:02.167695 27782 ts_tablet_manager.cc:1918] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:02.167951 27782 log.cc:1196] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/f27d4d43f75f470480cda6b00171bd33
I20260812 16:50:02.168637 27782 ts_tablet_manager.cc:1939] T f27d4d43f75f470480cda6b00171bd33 P d46ac544ce70420da317cafa90fe86aa: Deleting consensus metadata
I20260812 16:50:02.170976 26861 catalog_manager.cc:4841] TS d46ac544ce70420da317cafa90fe86aa (127.28.243.2:40383): tablet f27d4d43f75f470480cda6b00171bd33 (table test-table [id=16936aca82b64adfb4b655e9e1cdf25f]) successfully deleted
I20260812 16:50:02.174988 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:02.141275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.28.243.1:33265)
0812 16:50:02.141742 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:02.141750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:02.143849 (+ 2099us) server_negotiation.cc:407] Connection header received
0812 16:50:02.145026 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:02.145034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:02.145177 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:02.145276 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:02.153658 (+ 8382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.154901 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.157154 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.157840 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.157945 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:02.167978 (+ 10033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:02.168005 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:02.168028 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:02.168096 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:02.171430 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:02.172322 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:02.172330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:02.172333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:02.172497 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:02.172925 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:02.172929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:02.172931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:02.173384 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:02.173737 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:02.174307 (+ 570us) server_negotiation.cc:299] Negotiation successful
0812 16:50:02.174590 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260812 16:50:02.175813 27216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:02.304229 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 27000
I20260812 16:50:02.345355 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 27238
Aug 12 16:50:02 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:50:02.364243 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:02.346197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.28.243.2:40383)
0812 16:50:02.346399 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:02.346405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:02.346475 (+ 70us) server_negotiation.cc:407] Connection header received
0812 16:50:02.346676 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:02.346681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:02.346751 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:02.346848 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:02.348319 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.349250 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.350693 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.351259 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.351389 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:02.355973 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:02.355987 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:02.356004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:02.356053 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:02.360124 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:02.361237 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:02.361248 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:02.361250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:02.361449 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:02.361908 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:02.361912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:02.361914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:02.362420 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:02.363090 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:02.363537 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 16:50:02.363842 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:50:02.365190 27455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:02.488837 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 27238
I20260812 16:50:02.517153 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 27472
Aug 12 16:50:02 dist-test-slave-1vmg krb5kdc[26813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:50:02.535471 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:02.518043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.28.243.3:45063)
0812 16:50:02.518535 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:02.518539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:02.518557 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:50:02.518617 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:02.518621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:02.518700 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:02.518804 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:02.520373 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.521130 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.522883 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:02.523774 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:02.523931 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:02.529224 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:02.529245 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:02.529264 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:02.529311 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:02.532265 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:02.533221 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:02.533229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:02.533232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:02.533407 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:02.533771 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:02.533774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:02.533776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:02.534176 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:02.534521 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:02.534814 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 16:50:02.535092 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260812 16:50:02.536515 27697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:02.633541 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 27472
I20260812 16:50:02.665347 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 26835
I20260812 16:50:02.666976 26940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:02.786383 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 26835
2026-08-12T16:50:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5429 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T16:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:02Z Disabled control of system clock
I20260812 16:50:02.903070 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40341
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:50:03.319875 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:03.320237 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:03.320327 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:03.328464 27799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:03.328621 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:03.328693 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:03.328797 27799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:03.328872 27799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:03.339996 27799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:40341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:03.341483 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:03.343546 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:03.358525 27809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:03.358565 27804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:03.359370 27805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:03.360484 27799 server_base.cc:1034] running on GCE node
I20260812 16:50:03.361573 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:03.363274 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:03.364537 27799 hybrid_clock.cc:648] HybridClock initialized: now 1786553403364479 us; error 51 us; skew 500 ppm
I20260812 16:50:03.364984 27799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:03.368408 27799 webserver.cc:460] Webserver started at http://127.28.243.62:35215/ using document root <none> and password file <none>
I20260812 16:50:03.369262 27799 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:03.369375 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:03.369711 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:03.373096 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "f0e7d253cb3948c28ffa546b49d7c216"
format_stamp: "Formatted at 2026-08-12 16:50:03 on dist-test-slave-1vmg"
server_key: "651422a7b6f023b4c8ea5becd8b864fe"
server_key_iv: "82a3ca9f62979c824b49ae7663974b92"
server_key_version: "encryptionkey@0"
I20260812 16:50:03.374089 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "f0e7d253cb3948c28ffa546b49d7c216"
format_stamp: "Formatted at 2026-08-12 16:50:03 on dist-test-slave-1vmg"
server_key: "651422a7b6f023b4c8ea5becd8b864fe"
server_key_iv: "82a3ca9f62979c824b49ae7663974b92"
server_key_version: "encryptionkey@0"
I20260812 16:50:03.380398 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260812 16:50:03.384748 27820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:03.386533 27799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:50:03.386763 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "f0e7d253cb3948c28ffa546b49d7c216"
format_stamp: "Formatted at 2026-08-12 16:50:03 on dist-test-slave-1vmg"
server_key: "651422a7b6f023b4c8ea5becd8b864fe"
server_key_iv: "82a3ca9f62979c824b49ae7663974b92"
server_key_version: "encryptionkey@0"
I20260812 16:50:03.387050 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:03.416671 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:03.417673 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:03.417991 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:03.440807 27799 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:40341
I20260812 16:50:03.440793 27917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:40341 every 8 connection(s)
I20260812 16:50:03.442533 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:03.447153 27918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:03.450299 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 27799
I20260812 16:50:03.450533 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:03.451143 29644 external_mini_cluster.cc:1428] Setting key 4f3e088d9cda099ee2c071c6f2924ed4
I20260812 16:50:03.466429 27918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216: Bootstrap starting.
I20260812 16:50:03.468657 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:03.454268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.28.243.62:40341)
0812 16:50:03.455723 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:03.455737 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:03.455799 (+ 62us) server_negotiation.cc:407] Connection header received
0812 16:50:03.457833 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:03.457868 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:03.458499 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:03.458947 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:03.459852 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:03.461224 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:03.463373 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:03.464191 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:03.464362 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:03.465827 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:03.465837 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:03.465856 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:03.466609 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:03.466722 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:03.466752 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:03.466869 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:03.467152 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 16:50:03.467543 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":671,"threads_started":1}
I20260812 16:50:03.472666 27918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:03.474062 27918 log.cc:824] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:03.477531 27918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216: No bootstrap required, opened a new log
I20260812 16:50:03.486042 27918 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d253cb3948c28ffa546b49d7c216" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40341 } }
I20260812 16:50:03.487072 27918 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:03.487262 27918 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e7d253cb3948c28ffa546b49d7c216, State: Initialized, Role: FOLLOWER
I20260812 16:50:03.487958 27918 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d253cb3948c28ffa546b49d7c216" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40341 } }
I20260812 16:50:03.488260 27918 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:03.488420 27918 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:03.488632 27918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:03.491344 27918 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d253cb3948c28ffa546b49d7c216" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40341 } }
I20260812 16:50:03.491971 27918 leader_election.cc:302] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e7d253cb3948c28ffa546b49d7c216; no voters:
I20260812 16:50:03.492681 27918 leader_election.cc:288] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:03.492820 27926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:03.493577 27926 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [term 1 LEADER]: Becoming Leader. State: Replica: f0e7d253cb3948c28ffa546b49d7c216, State: Running, Role: LEADER
I20260812 16:50:03.494139 27926 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d253cb3948c28ffa546b49d7c216" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40341 } }
I20260812 16:50:03.494818 27918 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:03.496878 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e7d253cb3948c28ffa546b49d7c216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d253cb3948c28ffa546b49d7c216" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40341 } } }
I20260812 16:50:03.497514 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:03.497462 27929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e7d253cb3948c28ffa546b49d7c216. Latest consensus state: current_term: 1 leader_uuid: "f0e7d253cb3948c28ffa546b49d7c216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d253cb3948c28ffa546b49d7c216" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 40341 } } }
I20260812 16:50:03.497839 27929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:03.503597 27932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:03.508560 27932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:03.517932 27932 catalog_manager.cc:1347] Generated new cluster ID: 86267de16db74d02a6247155d7e0ff9e
I20260812 16:50:03.518065 27932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:03.529981 27932 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:03.531378 27932 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:03.540210 27932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216: Generated new TSK 0
I20260812 16:50:03.541196 27932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:03.545693 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40341
--builtin_ntp_servers=127.28.243.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:50:03.937206 27957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:03.937530 27957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:03.937626 27957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:03.946784 27957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:03.947060 27957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:03.947260 27957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:50:03.959431 27957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:03.960816 27957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:03.962782 27957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:03.979180 27965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:03.979391 27968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:03.979424 27966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:03.980459 27957 server_base.cc:1034] running on GCE node
I20260812 16:50:03.981349 27957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:03.982731 27957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:03.984023 27957 hybrid_clock.cc:648] HybridClock initialized: now 1786553403983947 us; error 70 us; skew 500 ppm
I20260812 16:50:03.984403 27957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:03.986869 27957 webserver.cc:460] Webserver started at http://127.28.243.1:43991/ using document root <none> and password file <none>
I20260812 16:50:03.987656 27957 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:03.987774 27957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:03.988102 27957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:03.991485 27957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "a447ad94e6034090bf922e9beb2a8687"
format_stamp: "Formatted at 2026-08-12 16:50:03 on dist-test-slave-1vmg"
server_key: "a8c9fef937c01d1527c094dc5c9ad3ff"
server_key_iv: "a6fd3d98c28e60d8d2afed7676cce1f4"
server_key_version: "encryptionkey@0"
I20260812 16:50:03.992470 27957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "a447ad94e6034090bf922e9beb2a8687"
format_stamp: "Formatted at 2026-08-12 16:50:03 on dist-test-slave-1vmg"
server_key: "a8c9fef937c01d1527c094dc5c9ad3ff"
server_key_iv: "a6fd3d98c28e60d8d2afed7676cce1f4"
server_key_version: "encryptionkey@0"
I20260812 16:50:03.998597 27957 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 16:50:04.002656 27979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:04.004236 27957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:50:04.004407 27957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "a447ad94e6034090bf922e9beb2a8687"
format_stamp: "Formatted at 2026-08-12 16:50:03 on dist-test-slave-1vmg"
server_key: "a8c9fef937c01d1527c094dc5c9ad3ff"
server_key_iv: "a6fd3d98c28e60d8d2afed7676cce1f4"
server_key_version: "encryptionkey@0"
I20260812 16:50:04.004634 27957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:04.027659 27957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:04.028650 27957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:04.028981 27957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:04.030077 27957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:04.031965 27957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:04.032078 27957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:04.032194 27957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:04.032267 27957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:04.068171 27957 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:44253
I20260812 16:50:04.068252 28179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:44253 every 8 connection(s)
I20260812 16:50:04.069547 27957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:50:04.073940 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 27957
I20260812 16:50:04.074141 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:50:04.074710 29644 external_mini_cluster.cc:1428] Setting key 82e3d4d31dea373f0deabef676b0f9d5
I20260812 16:50:04.078579 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40341
--builtin_ntp_servers=127.28.243.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:50:04.084741 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:04.072475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:57553 (local address 127.28.243.62:40341)
0812 16:50:04.072922 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:04.072927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:04.074452 (+ 1525us) server_negotiation.cc:407] Connection header received
0812 16:50:04.076818 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:04.076824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:04.076919 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:04.077016 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:04.078642 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.079611 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.081715 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.082150 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.082275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:04.083821 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:04.083826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:04.083829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:04.083853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:04.083896 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:04.083908 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:04.083989 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:04.084241 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 16:50:04.084356 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260812 16:50:04.085917 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:04.073038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40341 (local address 127.28.243.1:57553)
0812 16:50:04.074223 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:04.074299 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:04.076466 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:04.077319 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:04.077333 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:04.077471 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:04.078387 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:04.078418 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.079868 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.079874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:04.081507 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:04.081527 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.082427 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.082436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:04.082683 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:04.083458 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:04.083461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:04.083507 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:04.083606 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:04.084005 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:04.084019 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:04.084028 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:50:04.084214 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 16:50:04.084672 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":317,"threads_started":1}
I20260812 16:50:04.087584 28182 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40341
I20260812 16:50:04.087962 28182 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:04.088953 28182 heartbeater.cc:511] Master 127.28.243.62:40341 requested a full tablet report, sending...
I20260812 16:50:04.091598 27843 ts_manager.cc:194] Registered new tserver with Master: a447ad94e6034090bf922e9beb2a8687 (127.28.243.1:44253)
I20260812 16:50:04.093227 27843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.1:57553
W20260812 16:50:04.459156 28187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:04.459496 28187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:04.459589 28187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:04.467338 28187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:04.467461 28187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:04.467572 28187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:50:04.477296 28187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:04.478657 28187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:04.480669 28187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:04.495494 28203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:04.495605 28192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:04.495515 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:04.496600 28187 server_base.cc:1034] running on GCE node
I20260812 16:50:04.497735 28187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:04.499586 28187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:04.500944 28187 hybrid_clock.cc:648] HybridClock initialized: now 1786553404500876 us; error 107 us; skew 500 ppm
I20260812 16:50:04.501428 28187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:04.505039 28187 webserver.cc:460] Webserver started at http://127.28.243.2:44611/ using document root <none> and password file <none>
I20260812 16:50:04.505923 28187 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:04.506199 28187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:04.506709 28187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:04.510471 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "955ed684051e4efb92d7f8f580a91b4a"
format_stamp: "Formatted at 2026-08-12 16:50:04 on dist-test-slave-1vmg"
server_key: "68accaddcc90684fec691ea5a069e488"
server_key_iv: "9ea14b816548231598b782ceb324aa72"
server_key_version: "encryptionkey@0"
I20260812 16:50:04.511457 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "955ed684051e4efb92d7f8f580a91b4a"
format_stamp: "Formatted at 2026-08-12 16:50:04 on dist-test-slave-1vmg"
server_key: "68accaddcc90684fec691ea5a069e488"
server_key_iv: "9ea14b816548231598b782ceb324aa72"
server_key_version: "encryptionkey@0"
I20260812 16:50:04.518332 28187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:50:04.524432 28211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:04.526767 28187 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
I20260812 16:50:04.527096 28187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "955ed684051e4efb92d7f8f580a91b4a"
format_stamp: "Formatted at 2026-08-12 16:50:04 on dist-test-slave-1vmg"
server_key: "68accaddcc90684fec691ea5a069e488"
server_key_iv: "9ea14b816548231598b782ceb324aa72"
server_key_version: "encryptionkey@0"
I20260812 16:50:04.527364 28187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:04.552393 28187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:04.553376 28187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:04.553728 28187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:04.554868 28187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:04.556916 28187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:04.557027 28187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:04.557111 28187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:04.557173 28187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:04.595361 28187 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:39205
I20260812 16:50:04.595458 28393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:39205 every 8 connection(s)
I20260812 16:50:04.596876 28187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:50:04.599227 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 28187
I20260812 16:50:04.599429 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:50:04.600006 29644 external_mini_cluster.cc:1428] Setting key 4286e0f7e6ba4265c643348f8a43cea2
I20260812 16:50:04.603655 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40341
--builtin_ntp_servers=127.28.243.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:50:04.614833 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:04.600177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:51721 (local address 127.28.243.62:40341)
0812 16:50:04.600743 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:04.600750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:04.602256 (+ 1506us) server_negotiation.cc:407] Connection header received
0812 16:50:04.605110 (+ 2854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:04.605118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:04.605237 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:04.605409 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:04.607972 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.608859 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.611313 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.611877 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.612000 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:04.613952 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:04.613959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:04.613963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:04.613991 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:04.614037 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:04.614051 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:04.614134 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:04.614408 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 16:50:04.614513 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":300,"threads_started":1}
I20260812 16:50:04.616338 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:04.600901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40341 (local address 127.28.243.2:51721)
0812 16:50:04.602102 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:04.602155 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:04.604532 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:04.606022 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:04.606042 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:04.606235 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:04.607617 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:04.607656 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.609064 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.609070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:04.611108 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:04.611129 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:04.612148 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:04.612155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:04.612498 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:04.613398 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:04.613401 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:04.613465 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:04.613594 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:04.614159 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:04.614171 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:04.614179 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:50:04.614392 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 16:50:04.614868 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":364,"threads_started":1}
I20260812 16:50:04.617956 28394 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40341
I20260812 16:50:04.618273 28394 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:04.619225 28394 heartbeater.cc:511] Master 127.28.243.62:40341 requested a full tablet report, sending...
I20260812 16:50:04.621098 27843 ts_manager.cc:194] Registered new tserver with Master: 955ed684051e4efb92d7f8f580a91b4a (127.28.243.2:39205)
I20260812 16:50:04.621953 27843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.2:51721
W20260812 16:50:04.999342 28400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:04.999773 28400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:04.999927 28400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:05.008325 28400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:05.008487 28400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:05.008667 28400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:50:05.020035 28400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:05.021546 28400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:05.023638 28400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:05.042436 28406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:05.042510 28407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:05.042460 28411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:05.049173 28400 server_base.cc:1034] running on GCE node
I20260812 16:50:05.049973 28400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:05.051162 28400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:05.052440 28400 hybrid_clock.cc:648] HybridClock initialized: now 1786553405052374 us; error 55 us; skew 500 ppm
I20260812 16:50:05.052820 28400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:05.055868 28400 webserver.cc:460] Webserver started at http://127.28.243.3:35255/ using document root <none> and password file <none>
I20260812 16:50:05.056857 28400 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:05.057135 28400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:05.057504 28400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:05.060909 28400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "56ef98f1f8bb4066ada4bddaed17c779"
format_stamp: "Formatted at 2026-08-12 16:50:05 on dist-test-slave-1vmg"
server_key: "51603a7279268e82e348823411e1d22a"
server_key_iv: "ec4719442d4714633ea5b13750c37237"
server_key_version: "encryptionkey@0"
I20260812 16:50:05.061998 28400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "56ef98f1f8bb4066ada4bddaed17c779"
format_stamp: "Formatted at 2026-08-12 16:50:05 on dist-test-slave-1vmg"
server_key: "51603a7279268e82e348823411e1d22a"
server_key_iv: "ec4719442d4714633ea5b13750c37237"
server_key_version: "encryptionkey@0"
I20260812 16:50:05.068367 28400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 16:50:05.072487 28417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:05.073983 28400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:50:05.074173 28400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "56ef98f1f8bb4066ada4bddaed17c779"
format_stamp: "Formatted at 2026-08-12 16:50:05 on dist-test-slave-1vmg"
server_key: "51603a7279268e82e348823411e1d22a"
server_key_iv: "ec4719442d4714633ea5b13750c37237"
server_key_version: "encryptionkey@0"
I20260812 16:50:05.074406 28400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:05.093103 28400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:05.094026 28400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:05.094339 28400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:05.095518 28400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:05.097234 28182 heartbeater.cc:503] Master 127.28.243.62:40341 was elected leader, sending a full tablet report...
I20260812 16:50:05.098078 28400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:05.098256 28400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:05.098523 28400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:05.098647 28400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:05.144301 28400 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:35387
I20260812 16:50:05.144357 28607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:35387 every 8 connection(s)
I20260812 16:50:05.145871 28400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:50:05.147310 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 28400
I20260812 16:50:05.147516 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:50:05.148134 29644 external_mini_cluster.cc:1428] Setting key 7b4a1058530ca4a8c962a81e3bcbf800
I20260812 16:50:05.165383 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.151240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:35653 (local address 127.28.243.62:40341)
0812 16:50:05.151973 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.151979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.153651 (+ 1672us) server_negotiation.cc:407] Connection header received
0812 16:50:05.155952 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.155958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.156076 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.156196 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:05.158233 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.159214 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.161739 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.162176 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.162329 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.164072 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:05.164079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:05.164084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:05.164114 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:05.164167 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:05.164186 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:05.164296 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.164766 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.164962 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":427,"threads_started":1}
I20260812 16:50:05.167273 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.152050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:40341 (local address 127.28.243.3:35653)
0812 16:50:05.153431 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.153484 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.155313 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.156757 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.156772 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.156935 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:05.157876 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.157904 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.159482 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.159488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.161268 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.161295 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.162530 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.162537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.162806 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.163633 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:05.163636 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:05.163700 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:05.163806 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:05.164331 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:05.164346 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:05.164355 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.164670 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.165186 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":439,"threads_started":1}
I20260812 16:50:05.169597 28609 heartbeater.cc:347] Connected to a master server at 127.28.243.62:40341
I20260812 16:50:05.170485 28609 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:05.172258 28609 heartbeater.cc:511] Master 127.28.243.62:40341 requested a full tablet report, sending...
I20260812 16:50:05.174759 27843 ts_manager.cc:194] Registered new tserver with Master: 56ef98f1f8bb4066ada4bddaed17c779 (127.28.243.3:35387)
I20260812 16:50:05.176156 27843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.3:35653
I20260812 16:50:05.180706 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:50:05.194074 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.185332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.28.243.62:40341)
0812 16:50:05.185692 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.185699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.186107 (+ 408us) server_negotiation.cc:407] Connection header received
0812 16:50:05.186358 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.186363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.186537 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.186689 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:05.187551 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.188677 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.190678 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.191503 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.191657 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.192857 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:05.192863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:05.192867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:05.192892 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:05.193018 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:05.193050 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:05.193178 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.193512 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.193636 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260812 16:50:05.199314 27843 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:50:05.206786 27843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:50:05.254755 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.225331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.28.243.2:39205)
0812 16:50:05.245278 (+ 19947us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.245286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.245318 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:50:05.245431 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.245439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.245710 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.245950 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.247159 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.248911 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.251877 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.253185 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.253331 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.253631 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.254234 (+ 603us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.254424 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19749,"thread_start_us":19618,"threads_started":1}
I20260812 16:50:05.254763 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.224383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:39205 (local address 127.0.0.1:33072)
0812 16:50:05.238560 (+ 14177us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.238637 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.240391 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.246244 (+ 5853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.246250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.246284 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.246902 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.246948 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.249107 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.249113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.251598 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.251618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.253774 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.253781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.253982 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.254020 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.254136 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.254347 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13766,"mutex_wait_us":303,"thread_start_us":20143,"threads_started":1}
I20260812 16:50:05.257381 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.227245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.28.243.1:44253)
0812 16:50:05.243227 (+ 15982us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.243239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.244960 (+ 1721us) server_negotiation.cc:407] Connection header received
0812 16:50:05.245327 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.245340 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.245672 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.245944 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.247470 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.249081 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.252541 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.256026 (+ 3485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.256206 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.256518 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.256699 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.256990 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15714,"thread_start_us":15996,"threads_started":1}
I20260812 16:50:05.257591 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.226858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56510)
0812 16:50:05.244816 (+ 17958us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.244835 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.245085 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.245948 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.245955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.246184 (+ 229us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.247167 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.247186 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.249209 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.249213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.252225 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.252243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.256269 (+ 4026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.256276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.256454 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.256491 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.256597 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.256745 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17822,"mutex_wait_us":157,"thread_start_us":16816,"threads_started":1}
I20260812 16:50:05.258430 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.224422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:35387 (local address 127.0.0.1:36402)
0812 16:50:05.244041 (+ 19619us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.244066 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.244265 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.246609 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.246613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.246641 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.247284 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.247295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.249111 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.249116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.252225 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.252243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.256604 (+ 4361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.256610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.257115 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.257167 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.257329 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.257757 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19449,"mutex_wait_us":439,"thread_start_us":13517,"threads_started":1}
I20260812 16:50:05.259855 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.224786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.28.243.3:35387)
0812 16:50:05.245846 (+ 21060us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.245854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.245886 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:50:05.245992 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.245999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.246257 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.246480 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.247488 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.248919 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.252534 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.256042 (+ 3508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.256206 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.259111 (+ 2905us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.259203 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.259424 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20873,"thread_start_us":20698,"threads_started":1}
I20260812 16:50:05.259788 28289 tablet_service.cc:1467] Processing CreateTablet for tablet 9d64a54efc984529aef2fc52c7fc2971 (DEFAULT_TABLE table=test-table [id=b47df7380b8e4d04aa37a74675a345dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:05.261518 28289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d64a54efc984529aef2fc52c7fc2971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:05.262542 28067 tablet_service.cc:1467] Processing CreateTablet for tablet 9d64a54efc984529aef2fc52c7fc2971 (DEFAULT_TABLE table=test-table [id=b47df7380b8e4d04aa37a74675a345dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:05.262811 28502 tablet_service.cc:1467] Processing CreateTablet for tablet 9d64a54efc984529aef2fc52c7fc2971 (DEFAULT_TABLE table=test-table [id=b47df7380b8e4d04aa37a74675a345dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:05.264096 28502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d64a54efc984529aef2fc52c7fc2971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:05.264191 28067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d64a54efc984529aef2fc52c7fc2971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:05.279896 28645 tablet_bootstrap.cc:492] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Bootstrap starting.
I20260812 16:50:05.284741 28645 tablet_bootstrap.cc:654] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:05.286424 28645 log.cc:824] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:05.286845 28647 tablet_bootstrap.cc:492] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Bootstrap starting.
I20260812 16:50:05.287948 28646 tablet_bootstrap.cc:492] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Bootstrap starting.
I20260812 16:50:05.291496 28647 tablet_bootstrap.cc:654] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:05.292874 28645 tablet_bootstrap.cc:492] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: No bootstrap required, opened a new log
I20260812 16:50:05.293001 28647 log.cc:824] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:05.293236 28645 ts_tablet_manager.cc:1397] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260812 16:50:05.293483 28646 tablet_bootstrap.cc:654] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:05.294875 28646 log.cc:824] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:05.298213 28647 tablet_bootstrap.cc:492] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: No bootstrap required, opened a new log
I20260812 16:50:05.298537 28647 ts_tablet_manager.cc:1397] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260812 16:50:05.298672 28646 tablet_bootstrap.cc:492] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: No bootstrap required, opened a new log
I20260812 16:50:05.299261 28646 ts_tablet_manager.cc:1397] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260812 16:50:05.305461 28645 raft_consensus.cc:348] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.306185 28645 raft_consensus.cc:374] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:05.306304 28645 raft_consensus.cc:729] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ef98f1f8bb4066ada4bddaed17c779, State: Initialized, Role: FOLLOWER
I20260812 16:50:05.307168 28645 consensus_queue.cc:260] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.308172 28647 raft_consensus.cc:348] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.308947 28647 raft_consensus.cc:374] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:05.309077 28647 raft_consensus.cc:729] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a447ad94e6034090bf922e9beb2a8687, State: Initialized, Role: FOLLOWER
I20260812 16:50:05.310817 28609 heartbeater.cc:503] Master 127.28.243.62:40341 was elected leader, sending a full tablet report...
I20260812 16:50:05.310765 28647 consensus_queue.cc:260] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.311133 28646 raft_consensus.cc:348] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.311678 28646 raft_consensus.cc:374] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:05.311725 28645 ts_tablet_manager.cc:1428] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260812 16:50:05.311790 28646 raft_consensus.cc:729] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955ed684051e4efb92d7f8f580a91b4a, State: Initialized, Role: FOLLOWER
I20260812 16:50:05.312652 28646 consensus_queue.cc:260] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.314334 28647 ts_tablet_manager.cc:1428] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260812 16:50:05.316047 28394 heartbeater.cc:503] Master 127.28.243.62:40341 was elected leader, sending a full tablet report...
I20260812 16:50:05.317492 28646 ts_tablet_manager.cc:1428] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260812 16:50:05.322270 28183 tablet.cc:2369] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:50:05.349203 28395 tablet.cc:2369] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:50:05.398770 28613 tablet.cc:2369] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:05.526500 28658 raft_consensus.cc:482] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:05.526875 28658 raft_consensus.cc:504] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.528900 28658 leader_election.cc:288] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ef98f1f8bb4066ada4bddaed17c779 (127.28.243.3:35387), a447ad94e6034090bf922e9beb2a8687 (127.28.243.1:44253)
I20260812 16:50:05.530697 28654 raft_consensus.cc:482] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:05.531127 28654 raft_consensus.cc:504] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.533489 28654 leader_election.cc:288] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ef98f1f8bb4066ada4bddaed17c779 (127.28.243.3:35387), 955ed684051e4efb92d7f8f580a91b4a (127.28.243.2:39205)
I20260812 16:50:05.539686 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.529685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:59469 (local address 127.28.243.1:44253)
0812 16:50:05.529948 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.529956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.530742 (+ 786us) server_negotiation.cc:407] Connection header received
0812 16:50:05.531152 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.531159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.531272 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.531351 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.532295 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.533646 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.536885 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.538240 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.538420 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.538494 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.539175 (+ 681us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.539344 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 16:50:05.540510 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.529340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.28.243.2:59469)
0812 16:50:05.530586 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.530610 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.530904 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.531575 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.531579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.531608 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.532079 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.532104 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.533924 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.533931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.536547 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.536573 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.538722 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.538730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.538959 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.538995 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.539924 (+ 929us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.540105 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":1,"thread_start_us":812,"threads_started":1}
I20260812 16:50:05.543689 28098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "955ed684051e4efb92d7f8f580a91b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a447ad94e6034090bf922e9beb2a8687" is_pre_election: true
I20260812 16:50:05.544111 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.529676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:54947 (local address 127.28.243.3:35387)
0812 16:50:05.529916 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.529923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.531787 (+ 1864us) server_negotiation.cc:407] Connection header received
0812 16:50:05.532172 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.532180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.532446 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.532581 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.533248 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.537812 (+ 4564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.541621 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.542848 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.543009 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.543083 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.543701 (+ 618us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.543844 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:50:05.544351 28098 raft_consensus.cc:2454] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 955ed684051e4efb92d7f8f580a91b4a in term 0.
I20260812 16:50:05.544981 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.529668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:35387 (local address 127.28.243.2:54947)
0812 16:50:05.531646 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.531678 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.531845 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.532603 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.532608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.532635 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.533028 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.533041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.539123 (+ 6082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.539130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.541335 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.541355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.543296 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.543303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.543486 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.543522 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.544497 (+ 975us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.544665 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"thread_start_us":1033,"threads_started":1}
I20260812 16:50:05.545244 28214 leader_election.cc:302] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 955ed684051e4efb92d7f8f580a91b4a, a447ad94e6034090bf922e9beb2a8687; no voters:
I20260812 16:50:05.545791 28658 raft_consensus.cc:2781] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:05.545953 28658 raft_consensus.cc:482] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:05.546066 28658 raft_consensus.cc:3037] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:05.546142 28540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "955ed684051e4efb92d7f8f580a91b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ef98f1f8bb4066ada4bddaed17c779" is_pre_election: true
I20260812 16:50:05.546872 28540 raft_consensus.cc:2454] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 955ed684051e4efb92d7f8f580a91b4a in term 0.
I20260812 16:50:05.548713 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.533871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:35387 (local address 127.28.243.1:35465)
0812 16:50:05.538493 (+ 4622us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.538515 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.538644 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.539724 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.539730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.539780 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.540408 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.540423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.542445 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.542450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.545688 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.545705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.547919 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.547927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.548101 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.548143 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.548218 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.548365 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4470,"thread_start_us":4245,"threads_started":1}
I20260812 16:50:05.549355 28658 raft_consensus.cc:504] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.549726 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.534863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:45177 (local address 127.28.243.2:39205)
0812 16:50:05.535120 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.535126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.541833 (+ 6707us) server_negotiation.cc:407] Connection header received
0812 16:50:05.542087 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.542093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.542190 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.542251 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.543619 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.544619 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.547133 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.548427 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.548544 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.548607 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.549301 (+ 694us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.549451 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:50:05.549890 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.534415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:35465 (local address 127.28.243.3:35387)
0812 16:50:05.539277 (+ 4862us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.539282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.539305 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:50:05.539371 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.539376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.539464 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.540783 (+ 1319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:05.541072 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.542296 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.546096 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.547697 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.549276 (+ 1579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.549344 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.549394 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.549525 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4720,"thread_start_us":4473,"threads_started":1}
I20260812 16:50:05.550374 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.534028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:39205 (local address 127.28.243.1:45177)
0812 16:50:05.541727 (+ 7699us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.541749 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:05.541940 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:05.542435 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:05.542440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:05.542467 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:05.543412 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.543434 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.544817 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.544822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.546806 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:05.546823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.548791 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.548798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:05.548977 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.549111 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 16:50:05.549920 (+ 809us) client_negotiation.cc:240] Negotiation successful
0812 16:50:05.550094 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7560,"thread_start_us":8368,"threads_started":1}
I20260812 16:50:05.550527 28540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "a447ad94e6034090bf922e9beb2a8687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ef98f1f8bb4066ada4bddaed17c779" is_pre_election: true
I20260812 16:50:05.550549 28536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "955ed684051e4efb92d7f8f580a91b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ef98f1f8bb4066ada4bddaed17c779"
I20260812 16:50:05.550805 28540 raft_consensus.cc:2454] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a447ad94e6034090bf922e9beb2a8687 in term 0.
I20260812 16:50:05.550992 28658 leader_election.cc:288] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [CANDIDATE]: Term 1 election: Requested vote from peers 56ef98f1f8bb4066ada4bddaed17c779 (127.28.243.3:35387), a447ad94e6034090bf922e9beb2a8687 (127.28.243.1:44253)
I20260812 16:50:05.551443 27981 leader_election.cc:302] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ef98f1f8bb4066ada4bddaed17c779, a447ad94e6034090bf922e9beb2a8687; no voters:
I20260812 16:50:05.551470 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "a447ad94e6034090bf922e9beb2a8687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "955ed684051e4efb92d7f8f580a91b4a" is_pre_election: true
I20260812 16:50:05.551762 28098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "955ed684051e4efb92d7f8f580a91b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a447ad94e6034090bf922e9beb2a8687"
I20260812 16:50:05.551955 28654 raft_consensus.cc:2781] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:05.552030 28319 raft_consensus.cc:2379] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a447ad94e6034090bf922e9beb2a8687 in current term 1: Already voted for candidate 955ed684051e4efb92d7f8f580a91b4a in this term.
I20260812 16:50:05.552205 28098 raft_consensus.cc:3037] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:05.554837 28098 raft_consensus.cc:2454] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 955ed684051e4efb92d7f8f580a91b4a in term 1.
I20260812 16:50:05.555383 28214 leader_election.cc:302] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 955ed684051e4efb92d7f8f580a91b4a, a447ad94e6034090bf922e9beb2a8687; no voters: 56ef98f1f8bb4066ada4bddaed17c779
I20260812 16:50:05.555740 28658 raft_consensus.cc:2781] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:05.555940 28654 raft_consensus.cc:482] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:05.556082 28654 raft_consensus.cc:3037] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:50:05.556268 28658 raft_consensus.cc:686] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 LEADER]: Becoming Leader. State: Replica: 955ed684051e4efb92d7f8f580a91b4a, State: Running, Role: LEADER
I20260812 16:50:05.556843 28658 consensus_queue.cc:237] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.558466 28654 raft_consensus.cc:504] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.559864 28654 leader_election.cc:288] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [CANDIDATE]: Term 2 election: Requested vote from peers 56ef98f1f8bb4066ada4bddaed17c779 (127.28.243.3:35387), 955ed684051e4efb92d7f8f580a91b4a (127.28.243.2:39205)
I20260812 16:50:05.560488 28536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "a447ad94e6034090bf922e9beb2a8687" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ef98f1f8bb4066ada4bddaed17c779"
I20260812 16:50:05.560789 28536 raft_consensus.cc:3037] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 0 FOLLOWER]: Advancing to term 2
I20260812 16:50:05.560890 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d64a54efc984529aef2fc52c7fc2971" candidate_uuid: "a447ad94e6034090bf922e9beb2a8687" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "955ed684051e4efb92d7f8f580a91b4a"
I20260812 16:50:05.564157 28536 raft_consensus.cc:2454] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a447ad94e6034090bf922e9beb2a8687 in term 2.
I20260812 16:50:05.564898 27981 leader_election.cc:302] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56ef98f1f8bb4066ada4bddaed17c779, a447ad94e6034090bf922e9beb2a8687; no voters: 955ed684051e4efb92d7f8f580a91b4a
I20260812 16:50:05.565443 28654 raft_consensus.cc:2781] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:50:05.564877 27841 catalog_manager.cc:5495] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 955ed684051e4efb92d7f8f580a91b4a (127.28.243.2). New cstate: current_term: 1 leader_uuid: "955ed684051e4efb92d7f8f580a91b4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } health_report { overall_health: UNKNOWN } } }
I20260812 16:50:05.566123 28654 raft_consensus.cc:686] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 2 LEADER]: Becoming Leader. State: Replica: a447ad94e6034090bf922e9beb2a8687, State: Running, Role: LEADER
I20260812 16:50:05.566764 28654 consensus_queue.cc:237] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.573375 27841 catalog_manager.cc:5495] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 reported cstate change: term changed from 1 to 2, leader changed from 955ed684051e4efb92d7f8f580a91b4a (127.28.243.2) to a447ad94e6034090bf922e9beb2a8687 (127.28.243.1). New cstate: current_term: 2 leader_uuid: "a447ad94e6034090bf922e9beb2a8687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } health_report { overall_health: HEALTHY } } }
I20260812 16:50:05.630048 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.623224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.28.243.1:44253)
0812 16:50:05.623476 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.623482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.623500 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:50:05.623605 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.623609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.623711 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.623880 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:50:05.624278 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.625051 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.626992 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.627547 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.627694 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.628718 (+ 1024us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:50:05.629291 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:50:05.629447 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.629585 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.629702 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:50:05.640214 28319 raft_consensus.cc:3032] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 LEADER]: Stepping down as leader of term 1
I20260812 16:50:05.640368 28319 raft_consensus.cc:729] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 955ed684051e4efb92d7f8f580a91b4a, State: Running, Role: LEADER
I20260812 16:50:05.640411 28536 raft_consensus.cc:1261] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 2 FOLLOWER]: Refusing update from remote peer a447ad94e6034090bf922e9beb2a8687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 16:50:05.640690 28319 consensus_queue.cc:260] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 } } peers { permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 } } peers { permanent_uuid: "a447ad94e6034090bf922e9beb2a8687" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 44253 } }
I20260812 16:50:05.641188 28319 raft_consensus.cc:3037] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:50:05.641371 28672 consensus_queue.cc:1035] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ef98f1f8bb4066ada4bddaed17c779" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:05.644171 28319 raft_consensus.cc:1261] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 2 FOLLOWER]: Refusing update from remote peer a447ad94e6034090bf922e9beb2a8687: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 16:50:05.644965 28672 consensus_queue.cc:1035] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "955ed684051e4efb92d7f8f580a91b4a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:05.645241 28672 consensus_queue.cc:1225] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [LEADER]: Peer 955ed684051e4efb92d7f8f580a91b4a log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260812 16:50:05.647238 28319 pending_rounds.cc:76] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Aborting all ops after (but not including) 0
I20260812 16:50:05.647425 28319 pending_rounds.cc:97] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260812 16:50:05.647595 28319 raft_consensus.cc:2866] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 16:50:05.656008 28678 mvcc.cc:205] Tried to move back new op lower bound from 7317722749492064256 to 7317722749206876160. Current Snapshot: MvccSnapshot[applied={T|T < 7317722749492064256}]
I20260812 16:50:05.659206 28682 mvcc.cc:205] Tried to move back new op lower bound from 7317722749492064256 to 7317722749206876160. Current Snapshot: MvccSnapshot[applied={T|T < 7317722749492064256}]
I20260812 16:50:05.682605 27841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:05.683070 27841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:05.689414 27841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0e7d253cb3948c28ffa546b49d7c216: Sending DeleteTablet for 3 replicas of tablet 9d64a54efc984529aef2fc52c7fc2971
I20260812 16:50:05.690941 28502 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d64a54efc984529aef2fc52c7fc2971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:05 UTC) from {username='slave'} at 127.0.0.1:36402
I20260812 16:50:05.691514 28067 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d64a54efc984529aef2fc52c7fc2971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:05 UTC) from {username='slave'} at 127.0.0.1:56510
I20260812 16:50:05.691907 28289 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d64a54efc984529aef2fc52c7fc2971 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:05 UTC) from {username='slave'} at 127.0.0.1:33072
I20260812 16:50:05.692730 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 27957
I20260812 16:50:05.696833 28698 tablet_replica.cc:330] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: stopping tablet replica
I20260812 16:50:05.697419 28698 raft_consensus.cc:2229] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:05.697947 28698 raft_consensus.cc:2258] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:05.704731 28697 tablet_replica.cc:330] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: stopping tablet replica
I20260812 16:50:05.705176 28698 ts_tablet_manager.cc:1905] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:05.705687 28697 raft_consensus.cc:2229] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 2 LEADER]: Raft consensus shutting down.
I20260812 16:50:05.706645 28697 raft_consensus.cc:2258] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:05.707681 28693 tablet_replica.cc:330] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: stopping tablet replica
I20260812 16:50:05.708425 28693 raft_consensus.cc:2229] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:05.709105 28693 raft_consensus.cc:2258] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:05.713296 28697 ts_tablet_manager.cc:1905] T 9d64a54efc984529aef2fc52c7fc2971 P a447ad94e6034090bf922e9beb2a8687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:05.714376 28698 ts_tablet_manager.cc:1918] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260812 16:50:05.714622 28698 log.cc:1196] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/9d64a54efc984529aef2fc52c7fc2971
I20260812 16:50:05.715277 28698 ts_tablet_manager.cc:1939] T 9d64a54efc984529aef2fc52c7fc2971 P 955ed684051e4efb92d7f8f580a91b4a: Deleting consensus metadata
I20260812 16:50:05.715435 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.694043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.28.243.1:44253)
0812 16:50:05.694317 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.694324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.705640 (+ 11316us) server_negotiation.cc:407] Connection header received
0812 16:50:05.706258 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.706265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.706414 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.706505 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:05.707236 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.708441 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.710730 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.711565 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.711684 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.713389 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:05.713401 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:05.713423 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:05.714347 (+ 924us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:05.714445 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:05.714475 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:05.714584 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.714890 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.715045 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260812 16:50:05.717206 28174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:05.717993 28693 ts_tablet_manager.cc:1905] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:05.717957 27822 catalog_manager.cc:4841] TS 955ed684051e4efb92d7f8f580a91b4a (127.28.243.2:39205): tablet 9d64a54efc984529aef2fc52c7fc2971 (table test-table [id=b47df7380b8e4d04aa37a74675a345dd]) successfully deleted
I20260812 16:50:05.724810 28693 ts_tablet_manager.cc:1918] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260812 16:50:05.724984 28693 log.cc:1196] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/9d64a54efc984529aef2fc52c7fc2971
I20260812 16:50:05.725418 28693 ts_tablet_manager.cc:1939] T 9d64a54efc984529aef2fc52c7fc2971 P 56ef98f1f8bb4066ada4bddaed17c779: Deleting consensus metadata
I20260812 16:50:05.727500 27821 catalog_manager.cc:4841] TS 56ef98f1f8bb4066ada4bddaed17c779 (127.28.243.3:35387): tablet 9d64a54efc984529aef2fc52c7fc2971 (table test-table [id=b47df7380b8e4d04aa37a74675a345dd]) successfully deleted
I20260812 16:50:05.823474 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 27957
I20260812 16:50:05.865641 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 28187
W20260812 16:50:05.865566 27824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:44253 (error 108)
I20260812 16:50:05.868224 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.867294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56536)
0812 16:50:05.867817 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.868017 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":262,"threads_started":1}
W20260812 16:50:05.868625 27824 catalog_manager.cc:4584] TS a447ad94e6034090bf922e9beb2a8687 (127.28.243.1:44253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d64a54efc984529aef2fc52c7fc2971: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
I20260812 16:50:05.875422 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.866544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.28.243.2:39205)
0812 16:50:05.866729 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:05.866736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:05.866758 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:50:05.867246 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:05.867252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:05.867384 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:05.867517 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:05.868145 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.869599 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.871620 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:05.872315 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:05.872482 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:05.873750 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:05.873762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:05.873780 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:05.874458 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:05.874537 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:05.874560 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:05.874669 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:05.874815 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 16:50:05.875057 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:50:05.876183 28385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:05.923425 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.922582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56538)
0812 16:50:05.923044 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.923202 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260812 16:50:05.924974 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.924440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56548)
0812 16:50:05.924666 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.924759 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 16:50:05.974776 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 28187
I20260812 16:50:05.986264 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.985569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56556)
0812 16:50:05.985932 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.986054 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260812 16:50:05.987830 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:05.987219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56572)
0812 16:50:05.987489 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:05.987590 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260812 16:50:06.009829 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 28400
I20260812 16:50:06.018002 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:06.011566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.28.243.3:35387)
0812 16:50:06.011761 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:06.011766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:06.011784 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:50:06.011841 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:06.011845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:06.011957 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:06.012061 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:06.012498 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:06.013205 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:06.014556 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:06.015112 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:06.015236 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:06.016427 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:06.016436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:06.016452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:06.017188 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:06.017251 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:06.017275 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:06.017410 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:06.017586 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 16:50:06.017704 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:50:06.018735 28601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:06.068760 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:06.068233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56576)
0812 16:50:06.068491 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:06.068582 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260812 16:50:06.070112 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:06.069611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56588)
0812 16:50:06.069860 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:06.069954 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260812 16:50:06.115278 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 28400
I20260812 16:50:06.140671 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 27799
I20260812 16:50:06.142259 27907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:06.259868 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:06.259205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:44253 (local address 127.0.0.1:56596)
0812 16:50:06.259493 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:06.259640 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:44253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260812 16:50:06.259979 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 27799
2026-08-12T16:50:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3463 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:06 dist-test-slave-1vmg krb5kdc[28720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:06 dist-test-slave-1vmg krb5kdc[28720](info): set up 2 sockets
Aug 12 16:50:06 dist-test-slave-1vmg krb5kdc[28720](info): commencing operation
krb5kdc: starting...
W20260812 16:50:07.588419 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:07 dist-test-slave-1vmg krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:50:07.966853 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:41665
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:41665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:08.379871 28737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:08.380226 28737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:08.380340 28737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:08.388685 28737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:08.388842 28737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:08.388892 28737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:08.388955 28737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:08.389000 28737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:08.399739 28737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:41665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:41665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:08.401255 28737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:08.402963 28737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:08.417341 28750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:08.417387 28748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:08.417445 28746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:08.419605 28737 server_base.cc:1034] running on GCE node
I20260812 16:50:08.420655 28737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:08.422405 28737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:08.423669 28737 hybrid_clock.cc:648] HybridClock initialized: now 1786553408423586 us; error 84 us; skew 500 ppm
I20260812 16:50:08.424077 28737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:08 dist-test-slave-1vmg krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553408, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:08.437410 28737 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:50:08.439371 28737 webserver.cc:460] Webserver started at http://127.28.243.62:45295/ using document root <none> and password file <none>
I20260812 16:50:08.440220 28737 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:08.440364 28737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:08.440701 28737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:08.444234 28737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "cda5674e7e2e4650ba6939d5db4f3425"
format_stamp: "Formatted at 2026-08-12 16:50:08 on dist-test-slave-1vmg"
server_key: "81c9536f87108948b918e668f99b0447"
server_key_iv: "a65d8b9e782eb329274432d7bb1c9972"
server_key_version: "encryptionkey@0"
I20260812 16:50:08.445135 28737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "cda5674e7e2e4650ba6939d5db4f3425"
format_stamp: "Formatted at 2026-08-12 16:50:08 on dist-test-slave-1vmg"
server_key: "81c9536f87108948b918e668f99b0447"
server_key_iv: "a65d8b9e782eb329274432d7bb1c9972"
server_key_version: "encryptionkey@0"
I20260812 16:50:08.451319 28737 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:50:08.455964 28760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:08.457607 28737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:50:08.457829 28737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "cda5674e7e2e4650ba6939d5db4f3425"
format_stamp: "Formatted at 2026-08-12 16:50:08 on dist-test-slave-1vmg"
server_key: "81c9536f87108948b918e668f99b0447"
server_key_iv: "a65d8b9e782eb329274432d7bb1c9972"
server_key_version: "encryptionkey@0"
I20260812 16:50:08.458065 28737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:08.480832 28737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:08.487385 28737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:08.487700 28737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:08.509490 28737 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:41665
I20260812 16:50:08.509501 28835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:41665 every 8 connection(s)
I20260812 16:50:08.511127 28737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:08.515965 28836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:08.517432 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 28737
I20260812 16:50:08.517659 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:08.518328 29644 external_mini_cluster.cc:1428] Setting key abe37945ad3aa3629332cc42d3b12e6d
I20260812 16:50:08.533463 28836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425: Bootstrap starting.
Aug 12 16:50:08 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:08.537909 28836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:08.539199 28836 log.cc:824] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:08.543210 28836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425: No bootstrap required, opened a new log
I20260812 16:50:08.546864 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:08.520457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.28.243.62:41665)
0812 16:50:08.521333 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:08.521346 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:08.521394 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:50:08.523203 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:08.523230 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:08.523719 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:08.524099 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:08.526161 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:08.527714 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:08.530357 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:08.531402 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:08.531562 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:08.537087 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:08.537097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:08.537113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:08.537163 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:08.540891 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:08.542063 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:08.542071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:08.542078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:08.542294 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:08.542856 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:08.542860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:08.542863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:08.543548 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:08.543821 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:08.544650 (+ 829us) server_negotiation.cc:299] Negotiation successful
0812 16:50:08.545127 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":399,"threads_started":1}
I20260812 16:50:08.555843 28836 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5674e7e2e4650ba6939d5db4f3425" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41665 } }
I20260812 16:50:08.556486 28836 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:08.556605 28836 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda5674e7e2e4650ba6939d5db4f3425, State: Initialized, Role: FOLLOWER
I20260812 16:50:08.557322 28836 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5674e7e2e4650ba6939d5db4f3425" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41665 } }
I20260812 16:50:08.557629 28836 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:08.557775 28836 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:08.557935 28836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:08.560434 28836 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5674e7e2e4650ba6939d5db4f3425" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41665 } }
I20260812 16:50:08.561038 28836 leader_election.cc:302] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda5674e7e2e4650ba6939d5db4f3425; no voters:
I20260812 16:50:08.561806 28836 leader_election.cc:288] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:08.561988 28851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:08.562745 28851 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [term 1 LEADER]: Becoming Leader. State: Replica: cda5674e7e2e4650ba6939d5db4f3425, State: Running, Role: LEADER
I20260812 16:50:08.563512 28851 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5674e7e2e4650ba6939d5db4f3425" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41665 } }
I20260812 16:50:08.564301 28836 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:08.567169 28852 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda5674e7e2e4650ba6939d5db4f3425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5674e7e2e4650ba6939d5db4f3425" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41665 } } }
I20260812 16:50:08.567148 28853 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda5674e7e2e4650ba6939d5db4f3425. Latest consensus state: current_term: 1 leader_uuid: "cda5674e7e2e4650ba6939d5db4f3425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5674e7e2e4650ba6939d5db4f3425" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 41665 } } }
I20260812 16:50:08.567839 28852 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:08.567839 28853 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:08.573655 28863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:08.581156 28863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:08.594246 28863 catalog_manager.cc:1347] Generated new cluster ID: 3be50d61e49d4f83bca8448053b8ea4d
I20260812 16:50:08.594393 28863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:08.625784 28863 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:08.627950 28863 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:08.637929 28863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425: Generated new TSK 0
I20260812 16:50:08.638890 28863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:50:08.727608 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41665
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:09.124944 28893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:09.125269 28893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:09.125368 28893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:09.133265 28893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:09.133410 28893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:09.133531 28893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:50:09.147054 28893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:09.148566 28893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:09.150004 28893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:09.165201 28900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:09.165234 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:09.165128 28915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:09.166038 28893 server_base.cc:1034] running on GCE node
I20260812 16:50:09.166844 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:09.168617 28893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:09.170081 28893 hybrid_clock.cc:648] HybridClock initialized: now 1786553409169904 us; error 64 us; skew 500 ppm
I20260812 16:50:09.170619 28893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:09 dist-test-slave-1vmg krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:09.175896 28893 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:50:09.178013 28893 webserver.cc:460] Webserver started at http://127.28.243.1:44865/ using document root <none> and password file <none>
I20260812 16:50:09.178812 28893 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:09.178982 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:09.179348 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:09.182636 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9"
format_stamp: "Formatted at 2026-08-12 16:50:09 on dist-test-slave-1vmg"
server_key: "41689e41a8d8175d3e36159f29c69c91"
server_key_iv: "76b2c11b9abb3cad5068d3edc28df677"
server_key_version: "encryptionkey@0"
I20260812 16:50:09.183549 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9"
format_stamp: "Formatted at 2026-08-12 16:50:09 on dist-test-slave-1vmg"
server_key: "41689e41a8d8175d3e36159f29c69c91"
server_key_iv: "76b2c11b9abb3cad5068d3edc28df677"
server_key_version: "encryptionkey@0"
I20260812 16:50:09.190670 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 16:50:09.196409 28923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:09.198076 28893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260812 16:50:09.198309 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9"
format_stamp: "Formatted at 2026-08-12 16:50:09 on dist-test-slave-1vmg"
server_key: "41689e41a8d8175d3e36159f29c69c91"
server_key_iv: "76b2c11b9abb3cad5068d3edc28df677"
server_key_version: "encryptionkey@0"
I20260812 16:50:09.198547 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:09.232136 28893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:09.239640 28893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:09.239993 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:09.241279 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:09.242869 28893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:09.242986 28893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 16:50:09.243127 28893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:09.243182 28893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:09.283391 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:36807
I20260812 16:50:09.283449 29094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:36807 every 8 connection(s)
I20260812 16:50:09.284823 28893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:50:09.286403 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 28893
I20260812 16:50:09.286664 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:50:09.287410 29644 external_mini_cluster.cc:1428] Setting key 6b42b46b82f23d77141c3fb503ecb6bb
Aug 12 16:50:09 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:09.316051 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:09.289232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:40875 (local address 127.28.243.62:41665)
0812 16:50:09.289742 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:09.289754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:09.292271 (+ 2517us) server_negotiation.cc:407] Connection header received
0812 16:50:09.295433 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:09.295441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:09.295555 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:09.295709 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:09.297794 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.299052 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.301030 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.301686 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.301909 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:09.306789 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:09.306798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:09.306803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:09.306876 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:09.310442 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:09.312035 (+ 1593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:09.312042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:09.312045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:09.312291 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:09.313407 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:09.313412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:09.313414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:09.313816 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:09.314004 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:09.315253 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0812 16:50:09.315614 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
I20260812 16:50:09.317713 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:09.290431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:41665 (local address 127.28.243.1:40875)
0812 16:50:09.291750 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:09.291806 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:09.294787 (+ 2981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:09.296158 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:09.296176 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:09.296841 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:09.297551 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:09.297582 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.299327 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.299332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:09.300769 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:09.300789 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.302116 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.302122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:09.302396 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:09.303342 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:09.303370 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:09.306416 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:09.310723 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:09.310736 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:09.310758 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:09.311778 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:09.312535 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:09.312541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:09.312544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:09.313203 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:09.313982 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:09.313993 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:09.314507 (+ 514us) client_negotiation.cc:769] Sending connection context
0812 16:50:09.314819 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 16:50:09.315814 (+ 995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":587,"threads_started":1}
I20260812 16:50:09.319905 29095 heartbeater.cc:347] Connected to a master server at 127.28.243.62:41665
I20260812 16:50:09.320370 29095 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:09.321231 29095 heartbeater.cc:511] Master 127.28.243.62:41665 requested a full tablet report, sending...
I20260812 16:50:09.323838 28785 ts_manager.cc:194] Registered new tserver with Master: 4e55d9fc6e8c4d48bed2fe8edec717a9 (127.28.243.1:36807)
I20260812 16:50:09.325635 28785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:40875
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:50:09.377043 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41665
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:09.798741 29108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:09.799124 29108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:09.799239 29108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:09.807873 29108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:09.808028 29108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:09.808158 29108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:50:09.821126 29108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:09.822809 29108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:09.824339 29108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:09.840194 29115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:09.840514 29122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:09.840206 29116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:09.841133 29108 server_base.cc:1034] running on GCE node
I20260812 16:50:09.841854 29108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:09.843020 29108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:09.844296 29108 hybrid_clock.cc:648] HybridClock initialized: now 1786553409844234 us; error 50 us; skew 500 ppm
I20260812 16:50:09.844686 29108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:09 dist-test-slave-1vmg krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:09.848546 29108 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:50:09.853971 29108 webserver.cc:460] Webserver started at http://127.28.243.2:33049/ using document root <none> and password file <none>
I20260812 16:50:09.854903 29108 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:09.855384 29108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:09.855929 29108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:09.860014 29108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "d44ee299399248b3ab9beeb7b001b820"
format_stamp: "Formatted at 2026-08-12 16:50:09 on dist-test-slave-1vmg"
server_key: "116b708f7c426e4e10d9e5f4c84a686f"
server_key_iv: "75fa34c2a9ad3fb65eec8f543ce1c2f3"
server_key_version: "encryptionkey@0"
I20260812 16:50:09.861387 29108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "d44ee299399248b3ab9beeb7b001b820"
format_stamp: "Formatted at 2026-08-12 16:50:09 on dist-test-slave-1vmg"
server_key: "116b708f7c426e4e10d9e5f4c84a686f"
server_key_iv: "75fa34c2a9ad3fb65eec8f543ce1c2f3"
server_key_version: "encryptionkey@0"
I20260812 16:50:09.869071 29108 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260812 16:50:09.874423 29131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:09.876632 29108 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 16:50:09.876860 29108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "d44ee299399248b3ab9beeb7b001b820"
format_stamp: "Formatted at 2026-08-12 16:50:09 on dist-test-slave-1vmg"
server_key: "116b708f7c426e4e10d9e5f4c84a686f"
server_key_iv: "75fa34c2a9ad3fb65eec8f543ce1c2f3"
server_key_version: "encryptionkey@0"
I20260812 16:50:09.877100 29108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:09.894161 29108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:09.900239 29108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:09.900578 29108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:09.901710 29108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:09.903386 29108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:09.903478 29108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:09.903606 29108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:09.903668 29108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:09.944511 29108 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:40197
I20260812 16:50:09.944576 29327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:40197 every 8 connection(s)
I20260812 16:50:09.945974 29108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:50:09.946601 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 29108
I20260812 16:50:09.946812 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:50:09.947728 29644 external_mini_cluster.cc:1428] Setting key 3b415aa5566844643af3cfdee2604245
Aug 12 16:50:09 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:09.976610 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:09.949601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:55379 (local address 127.28.243.62:41665)
0812 16:50:09.950077 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:09.950082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:09.952449 (+ 2367us) server_negotiation.cc:407] Connection header received
0812 16:50:09.955372 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:09.955377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:09.955610 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:09.956082 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:09.958568 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.959457 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.962407 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.963300 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.963482 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:09.968655 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:09.968664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:09.968668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:09.968717 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:09.971629 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:09.973262 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:09.973268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:09.973271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:09.973467 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:09.974114 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:09.974119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:09.974122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:09.974531 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:09.974690 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:09.976079 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0812 16:50:09.976315 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
I20260812 16:50:09.978233 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:09.950816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:41665 (local address 127.28.243.2:55379)
0812 16:50:09.952159 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:09.952257 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:09.954819 (+ 2562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:09.956413 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:09.956427 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:09.957291 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:09.958221 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:09.958261 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.959649 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.959654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:09.962166 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:09.962192 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:09.963638 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:09.963647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:09.964035 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:09.964918 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:09.964954 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:09.968248 (+ 3294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:09.971884 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:09.971897 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:09.971920 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:09.973058 (+ 1138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:09.973644 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:09.973649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:09.973652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:09.973972 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:09.974703 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:09.974715 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:09.975611 (+ 896us) client_negotiation.cc:769] Sending connection context
0812 16:50:09.976003 (+ 392us) client_negotiation.cc:240] Negotiation successful
0812 16:50:09.976662 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":471,"threads_started":1}
I20260812 16:50:09.980114 29328 heartbeater.cc:347] Connected to a master server at 127.28.243.62:41665
I20260812 16:50:09.980680 29328 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:09.981722 29328 heartbeater.cc:511] Master 127.28.243.62:41665 requested a full tablet report, sending...
I20260812 16:50:09.983620 28785 ts_manager.cc:194] Registered new tserver with Master: d44ee299399248b3ab9beeb7b001b820 (127.28.243.2:40197)
I20260812 16:50:09.984555 28785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:55379
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:50:10.038098 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41665
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:50:10.330610 29095 heartbeater.cc:503] Master 127.28.243.62:41665 was elected leader, sending a full tablet report...
W20260812 16:50:10.432932 29346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:10.433281 29346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:10.433421 29346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:10.441866 29346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:10.442314 29346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:10.442528 29346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:50:10.455027 29346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:41665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:10.456566 29346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:10.457998 29346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:10.473317 29354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:10.473295 29357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:10.473392 29355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:10.474826 29346 server_base.cc:1034] running on GCE node
I20260812 16:50:10.475795 29346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:10.477169 29346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:10.478490 29346 hybrid_clock.cc:648] HybridClock initialized: now 1786553410478421 us; error 57 us; skew 500 ppm
I20260812 16:50:10.478842 29346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:10 dist-test-slave-1vmg krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:10.482837 29346 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:50:10.484695 29346 webserver.cc:460] Webserver started at http://127.28.243.3:41397/ using document root <none> and password file <none>
I20260812 16:50:10.485497 29346 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:10.485656 29346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:10.486011 29346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:10.489445 29346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce"
format_stamp: "Formatted at 2026-08-12 16:50:10 on dist-test-slave-1vmg"
server_key: "90f5408535552320f706f6bdca244cc4"
server_key_iv: "06408a0a30de9d938f2de7345282b646"
server_key_version: "encryptionkey@0"
I20260812 16:50:10.490528 29346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce"
format_stamp: "Formatted at 2026-08-12 16:50:10 on dist-test-slave-1vmg"
server_key: "90f5408535552320f706f6bdca244cc4"
server_key_iv: "06408a0a30de9d938f2de7345282b646"
server_key_version: "encryptionkey@0"
I20260812 16:50:10.498497 29346 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 16:50:10.503895 29369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:10.505815 29346 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:50:10.506039 29346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce"
format_stamp: "Formatted at 2026-08-12 16:50:10 on dist-test-slave-1vmg"
server_key: "90f5408535552320f706f6bdca244cc4"
server_key_iv: "06408a0a30de9d938f2de7345282b646"
server_key_version: "encryptionkey@0"
I20260812 16:50:10.506279 29346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:10.532438 29346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:10.538280 29346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:10.538638 29346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:10.539783 29346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:10.541374 29346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:10.541465 29346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:10.541586 29346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:10.541646 29346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:10.581825 29346 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:41399
I20260812 16:50:10.581877 29554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:41399 every 8 connection(s)
I20260812 16:50:10.583400 29346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:50:10.592284 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 29346
I20260812 16:50:10.592741 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:50:10.593410 29644 external_mini_cluster.cc:1428] Setting key badf6aaf1f7f090add2cdc97e00e66ee
Aug 12 16:50:10 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:10.608047 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.586377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:52677 (local address 127.28.243.62:41665)
0812 16:50:10.586838 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.586843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.587980 (+ 1137us) server_negotiation.cc:407] Connection header received
0812 16:50:10.590022 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.590045 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.590177 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.590301 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:10.592162 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.593005 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.594763 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.595332 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.595454 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.599460 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:10.599465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:10.599469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:10.599508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:10.603278 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:10.604670 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:10.604677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:10.604679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:10.605081 (+ 402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:10.605801 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:10.605805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:10.605808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:10.606141 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:10.606320 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.607308 (+ 988us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.607611 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260812 16:50:10.609172 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.587018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:41665 (local address 127.28.243.3:52677)
0812 16:50:10.587804 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:10.587855 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:10.589613 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:10.590590 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:10.590600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:10.591267 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:10.591942 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.591964 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.593180 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.593184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.594577 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.594600 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.595582 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.595589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.595849 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.596570 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:10.596596 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:10.599174 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:10.603703 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:10.603716 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:10.603740 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:10.604469 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:10.605322 (+ 853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:10.605328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:10.605331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:10.605640 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:10.606295 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:10.606304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:10.606714 (+ 410us) client_negotiation.cc:769] Sending connection context
0812 16:50:10.607157 (+ 443us) client_negotiation.cc:240] Negotiation successful
0812 16:50:10.607828 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":337,"threads_started":1}
I20260812 16:50:10.610678 29556 heartbeater.cc:347] Connected to a master server at 127.28.243.62:41665
I20260812 16:50:10.611097 29556 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:10.611992 29556 heartbeater.cc:511] Master 127.28.243.62:41665 requested a full tablet report, sending...
I20260812 16:50:10.613946 28785 ts_manager.cc:194] Registered new tserver with Master: 6a7d036cb8ff4cf58b5d2f369a26c3ce (127.28.243.3:41399)
I20260812 16:50:10.615075 28785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:52677
I20260812 16:50:10.624521 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:50:10.647122 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.629805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.28.243.62:41665)
0812 16:50:10.630212 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.630220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.630518 (+ 298us) server_negotiation.cc:407] Connection header received
0812 16:50:10.630766 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.630772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.630867 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.631066 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:10.633259 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.634040 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.636185 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.636717 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.636880 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.641034 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:10.641041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:10.641045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:10.641108 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:10.643957 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:10.644974 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:10.644978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:10.644980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:10.645124 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:10.645575 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:10.645581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:10.645583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:10.645923 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:10.646202 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.646571 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.646775 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 16:50:10.651350 28785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:50:10.658412 28785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:50:10.699268 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.672724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:36807 (local address 127.0.0.1:49504)
0812 16:50:10.673482 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:10.673532 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:10.673728 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:10.685200 (+ 11472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:10.685210 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:10.685244 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:10.685794 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.685811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.687829 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.687838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.691649 (+ 3811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.691665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.697847 (+ 6182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.697854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.698102 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.698159 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 16:50:10.698309 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 16:50:10.698546 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"spinlock_wait_cycles":8320,"thread_start_us":473,"threads_started":1}
I20260812 16:50:10.699522 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.673299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.28.243.2:40197)
0812 16:50:10.684488 (+ 11189us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.684495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.685290 (+ 795us) server_negotiation.cc:407] Connection header received
0812 16:50:10.685619 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.685628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.685899 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.686076 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:10.689290 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.691501 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.694199 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.697896 (+ 3697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.698061 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.698376 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.698965 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.699191 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11029,"thread_start_us":11205,"threads_started":1}
I20260812 16:50:10.699738 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.673122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.28.243.1:36807)
0812 16:50:10.684115 (+ 10993us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.684126 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.684166 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:50:10.684290 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.684298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.684628 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.684888 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:10.686013 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.687587 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.691848 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.697213 (+ 5365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.697381 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.697671 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.699085 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.699381 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10803,"thread_start_us":10668,"threads_started":1}
I20260812 16:50:10.700318 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.675190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:41399 (local address 127.0.0.1:44436)
0812 16:50:10.690453 (+ 15263us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:10.690478 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:10.690641 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:10.691891 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:10.691896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:10.691932 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:10.692384 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.692398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.694397 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.694402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.696383 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.696403 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.699004 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.699013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.699302 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.699337 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:50:10.699438 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 16:50:10.699577 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15048,"thread_start_us":14792,"threads_started":1}
I20260812 16:50:10.700430 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.673174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:40197 (local address 127.0.0.1:38580)
0812 16:50:10.685165 (+ 11991us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:10.685184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:10.685415 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:10.686460 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:10.686465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:10.686493 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:10.688874 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.688893 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.691674 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.691680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.693876 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.693896 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.698532 (+ 4636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.698538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.698714 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.698744 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 16:50:10.699866 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0812 16:50:10.700014 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11845,"mutex_wait_us":637,"spinlock_wait_cycles":1792,"thread_start_us":11031,"threads_started":1}
I20260812 16:50:10.701316 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.675937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.28.243.3:41399)
0812 16:50:10.691312 (+ 15375us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.691320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.691348 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:50:10.691435 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.691441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.691671 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.691909 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:10.692583 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.694217 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.696615 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.698231 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.698388 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.698597 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.700205 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.700949 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15175,"thread_start_us":15015,"threads_started":1}
I20260812 16:50:10.704375 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 30f22357b2b3496995cfbcca66429367 (DEFAULT_TABLE table=test-table [id=42f6c958389d4a43b5a23d4326d9a5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:10.704440 28994 tablet_service.cc:1467] Processing CreateTablet for tablet 30f22357b2b3496995cfbcca66429367 (DEFAULT_TABLE table=test-table [id=42f6c958389d4a43b5a23d4326d9a5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:10.705117 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 30f22357b2b3496995cfbcca66429367 (DEFAULT_TABLE table=test-table [id=42f6c958389d4a43b5a23d4326d9a5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:10.705940 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f22357b2b3496995cfbcca66429367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:10.706024 28994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f22357b2b3496995cfbcca66429367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:10.706625 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f22357b2b3496995cfbcca66429367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:10.722127 29589 tablet_bootstrap.cc:492] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Bootstrap starting.
I20260812 16:50:10.725383 29590 tablet_bootstrap.cc:492] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Bootstrap starting.
I20260812 16:50:10.726390 29589 tablet_bootstrap.cc:654] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:10.726769 29591 tablet_bootstrap.cc:492] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Bootstrap starting.
I20260812 16:50:10.728065 29589 log.cc:824] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:10.730865 29591 tablet_bootstrap.cc:654] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:10.732398 29591 log.cc:824] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:10.733136 29590 tablet_bootstrap.cc:654] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:10.734722 29590 log.cc:824] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:10.735330 29589 tablet_bootstrap.cc:492] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: No bootstrap required, opened a new log
I20260812 16:50:10.735761 29589 ts_tablet_manager.cc:1397] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260812 16:50:10.738567 29591 tablet_bootstrap.cc:492] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: No bootstrap required, opened a new log
I20260812 16:50:10.738567 29590 tablet_bootstrap.cc:492] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: No bootstrap required, opened a new log
I20260812 16:50:10.738901 29591 ts_tablet_manager.cc:1397] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260812 16:50:10.738901 29590 ts_tablet_manager.cc:1397] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260812 16:50:10.743551 29589 raft_consensus.cc:348] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.744005 29589 raft_consensus.cc:374] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:10.744124 29589 raft_consensus.cc:729] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44ee299399248b3ab9beeb7b001b820, State: Initialized, Role: FOLLOWER
I20260812 16:50:10.744961 29589 consensus_queue.cc:260] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.747268 29589 ts_tablet_manager.cc:1428] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260812 16:50:10.748137 29328 heartbeater.cc:503] Master 127.28.243.62:41665 was elected leader, sending a full tablet report...
I20260812 16:50:10.751026 29591 raft_consensus.cc:348] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.751231 29590 raft_consensus.cc:348] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.751591 29591 raft_consensus.cc:374] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:10.751673 29590 raft_consensus.cc:374] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:10.751731 29591 raft_consensus.cc:729] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e55d9fc6e8c4d48bed2fe8edec717a9, State: Initialized, Role: FOLLOWER
I20260812 16:50:10.751804 29590 raft_consensus.cc:729] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d036cb8ff4cf58b5d2f369a26c3ce, State: Initialized, Role: FOLLOWER
I20260812 16:50:10.752460 29590 consensus_queue.cc:260] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.752461 29591 consensus_queue.cc:260] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.754889 29556 heartbeater.cc:503] Master 127.28.243.62:41665 was elected leader, sending a full tablet report...
I20260812 16:50:10.755307 29590 ts_tablet_manager.cc:1428] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260812 16:50:10.755586 29591 ts_tablet_manager.cc:1428] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
I20260812 16:50:10.758226 29602 raft_consensus.cc:482] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:10.758524 29602 raft_consensus.cc:504] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.764050 29602 leader_election.cc:288] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e55d9fc6e8c4d48bed2fe8edec717a9 (127.28.243.1:36807), d44ee299399248b3ab9beeb7b001b820 (127.28.243.2:40197)
I20260812 16:50:10.770015 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.763161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:60309 (local address 127.28.243.1:36807)
0812 16:50:10.763342 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.763348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.763367 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:50:10.763439 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.763443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.763523 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.763625 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:10.764358 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.765152 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.767712 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.768803 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.768965 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.769026 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.769572 (+ 546us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.769735 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 16:50:10.770186 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.762071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:36807 (local address 127.28.243.3:60309)
0812 16:50:10.762581 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:10.762615 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:10.762768 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:10.763676 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:10.763681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:10.763712 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:10.763971 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.763985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.765305 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.765311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.767459 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.767492 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.769105 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.769112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.769331 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.769373 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:50:10.769501 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 16:50:10.769691 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":4}
I20260812 16:50:10.770799 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.762590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:40197 (local address 127.28.243.3:48541)
0812 16:50:10.763746 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:10.763768 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:10.764009 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:10.764503 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:10.764507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:10.764527 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:10.764828 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.764837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.766259 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.766263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.767962 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:10.767982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.769699 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.769706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:10.769893 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.769925 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:50:10.770042 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 16:50:10.770197 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":1120,"threads_started":1}
I20260812 16:50:10.770800 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.762619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:48541 (local address 127.28.243.2:40197)
0812 16:50:10.762867 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.762873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.763856 (+ 983us) server_negotiation.cc:407] Connection header received
0812 16:50:10.764162 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.764167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.764246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.764335 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:10.764988 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.766052 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.768196 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.769445 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.769595 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.769664 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.770181 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.770404 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 16:50:10.772173 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f22357b2b3496995cfbcca66429367" candidate_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44ee299399248b3ab9beeb7b001b820" is_pre_election: true
I20260812 16:50:10.772179 29021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f22357b2b3496995cfbcca66429367" candidate_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" is_pre_election: true
I20260812 16:50:10.772851 29245 raft_consensus.cc:2454] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d036cb8ff4cf58b5d2f369a26c3ce in term 0.
I20260812 16:50:10.772851 29021 raft_consensus.cc:2454] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d036cb8ff4cf58b5d2f369a26c3ce in term 0.
I20260812 16:50:10.773782 29371 leader_election.cc:302] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e55d9fc6e8c4d48bed2fe8edec717a9, 6a7d036cb8ff4cf58b5d2f369a26c3ce; no voters:
I20260812 16:50:10.774370 29602 raft_consensus.cc:2781] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:10.774570 29602 raft_consensus.cc:482] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:10.774684 29602 raft_consensus.cc:3037] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:10.777063 29602 raft_consensus.cc:504] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
I20260812 16:50:10.777746 29602 leader_election.cc:288] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [CANDIDATE]: Term 1 election: Requested vote from peers 4e55d9fc6e8c4d48bed2fe8edec717a9 (127.28.243.1:36807), d44ee299399248b3ab9beeb7b001b820 (127.28.243.2:40197)
I20260812 16:50:10.778435 29245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f22357b2b3496995cfbcca66429367" candidate_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44ee299399248b3ab9beeb7b001b820"
I20260812 16:50:10.778432 29021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f22357b2b3496995cfbcca66429367" candidate_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9"
I20260812 16:50:10.778673 29021 raft_consensus.cc:3037] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:10.778673 29245 raft_consensus.cc:3037] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:10.782089 29245 raft_consensus.cc:2454] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d036cb8ff4cf58b5d2f369a26c3ce in term 1.
I20260812 16:50:10.782089 29021 raft_consensus.cc:2454] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d036cb8ff4cf58b5d2f369a26c3ce in term 1.
I20260812 16:50:10.782747 29371 leader_election.cc:302] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e55d9fc6e8c4d48bed2fe8edec717a9, 6a7d036cb8ff4cf58b5d2f369a26c3ce; no voters:
I20260812 16:50:10.783182 29602 raft_consensus.cc:2781] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:10.783752 29602 raft_consensus.cc:686] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 1 LEADER]: Becoming Leader. State: Replica: 6a7d036cb8ff4cf58b5d2f369a26c3ce, State: Running, Role: LEADER
I20260812 16:50:10.784273 29602 consensus_queue.cc:237] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } }
W20260812 16:50:10.788269 29100 tablet.cc:2369] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:10.789849 28785 catalog_manager.cc:5495] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7d036cb8ff4cf58b5d2f369a26c3ce (127.28.243.3). New cstate: current_term: 1 leader_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7d036cb8ff4cf58b5d2f369a26c3ce" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 41399 } health_report { overall_health: HEALTHY } } }
I20260812 16:50:10.824496 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.817450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.28.243.3:41399)
0812 16:50:10.817684 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.817690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.817711 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:50:10.817946 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.817950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.818046 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.818242 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:50:10.818672 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.819440 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.821178 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.822215 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.822342 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.823112 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:50:10.823672 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:50:10.823825 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.824136 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.824186 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:50:10.835294 29245 raft_consensus.cc:1261] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 1 FOLLOWER]: Refusing update from remote peer 6a7d036cb8ff4cf58b5d2f369a26c3ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 16:50:10.835386 29557 tablet.cc:2369] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:10.836447 29021 raft_consensus.cc:1261] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 1 FOLLOWER]: Refusing update from remote peer 6a7d036cb8ff4cf58b5d2f369a26c3ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:10.836611 29608 consensus_queue.cc:1035] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44ee299399248b3ab9beeb7b001b820" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:10.837774 29608 consensus_queue.cc:1035] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e55d9fc6e8c4d48bed2fe8edec717a9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:10.873878 28785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:10.874184 28785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:10.885337 28785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda5674e7e2e4650ba6939d5db4f3425: Sending DeleteTablet for 3 replicas of tablet 30f22357b2b3496995cfbcca66429367
I20260812 16:50:10.886885 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f22357b2b3496995cfbcca66429367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:10 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:49504
I20260812 16:50:10.887336 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f22357b2b3496995cfbcca66429367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:10 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:44436
I20260812 16:50:10.887328 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f22357b2b3496995cfbcca66429367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:10 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:38580
I20260812 16:50:10.887986 29632 tablet_replica.cc:330] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: stopping tablet replica
I20260812 16:50:10.888561 29632 raft_consensus.cc:2229] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:10.889173 29632 raft_consensus.cc:2258] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:10.889724 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 28893
I20260812 16:50:10.894294 29632 ts_tablet_manager.cc:1905] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:10.903663 29632 ts_tablet_manager.cc:1918] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:10.903945 29632 log.cc:1196] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/30f22357b2b3496995cfbcca66429367
I20260812 16:50:10.903679 29634 tablet_replica.cc:330] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: stopping tablet replica
I20260812 16:50:10.904593 29632 ts_tablet_manager.cc:1939] T 30f22357b2b3496995cfbcca66429367 P 4e55d9fc6e8c4d48bed2fe8edec717a9: Deleting consensus metadata
I20260812 16:50:10.905275 29634 raft_consensus.cc:2229] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:50:10.907364 28762 catalog_manager.cc:4841] TS 4e55d9fc6e8c4d48bed2fe8edec717a9 (127.28.243.1:36807): tablet 30f22357b2b3496995cfbcca66429367 (table test-table [id=42f6c958389d4a43b5a23d4326d9a5aa]) successfully deleted
I20260812 16:50:10.908286 29634 raft_consensus.cc:2258] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:10.910712 29633 tablet_replica.cc:330] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: stopping tablet replica
I20260812 16:50:10.911342 29633 raft_consensus.cc:2229] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:10.911948 29633 raft_consensus.cc:2258] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:10.914682 29633 ts_tablet_manager.cc:1905] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:10.914702 29634 ts_tablet_manager.cc:1905] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 16:50:10 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:50:10.922851 29633 ts_tablet_manager.cc:1918] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:10.923177 29633 log.cc:1196] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/30f22357b2b3496995cfbcca66429367
I20260812 16:50:10.923828 29633 ts_tablet_manager.cc:1939] T 30f22357b2b3496995cfbcca66429367 P d44ee299399248b3ab9beeb7b001b820: Deleting consensus metadata
I20260812 16:50:10.924214 29634 ts_tablet_manager.cc:1918] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:10.924515 29634 log.cc:1196] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/30f22357b2b3496995cfbcca66429367
I20260812 16:50:10.925189 29634 ts_tablet_manager.cc:1939] T 30f22357b2b3496995cfbcca66429367 P 6a7d036cb8ff4cf58b5d2f369a26c3ce: Deleting consensus metadata
I20260812 16:50:10.927095 28767 catalog_manager.cc:4841] TS d44ee299399248b3ab9beeb7b001b820 (127.28.243.2:40197): tablet 30f22357b2b3496995cfbcca66429367 (table test-table [id=42f6c958389d4a43b5a23d4326d9a5aa]) successfully deleted
I20260812 16:50:10.928015 28766 catalog_manager.cc:4841] TS 6a7d036cb8ff4cf58b5d2f369a26c3ce (127.28.243.3:41399): tablet 30f22357b2b3496995cfbcca66429367 (table test-table [id=42f6c958389d4a43b5a23d4326d9a5aa]) successfully deleted
I20260812 16:50:10.937058 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:10.890830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.28.243.1:36807)
0812 16:50:10.891030 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:10.891037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:10.898908 (+ 7871us) server_negotiation.cc:407] Connection header received
0812 16:50:10.900087 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:10.900093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:10.900211 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:10.900317 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:10.906044 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.908439 (+ 2395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.911748 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:10.912436 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:10.912602 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:10.926842 (+ 14240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:10.926869 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:10.926896 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:10.926999 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:10.931554 (+ 4555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:10.933283 (+ 1729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:10.933298 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:10.933303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:10.933595 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:10.934471 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:10.934477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:10.934481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:10.935155 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:10.935533 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:10.936309 (+ 776us) server_negotiation.cc:299] Negotiation successful
0812 16:50:10.936604 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:50:10.938051 29090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:11.112879 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 28893
I20260812 16:50:11.151412 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 29108
Aug 12 16:50:11 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:50:11.175926 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:11.152439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.28.243.2:40197)
0812 16:50:11.153112 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:11.153118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:11.153137 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:50:11.153204 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:11.153210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:11.153303 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:11.153412 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:11.155320 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:11.156462 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:11.158596 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:11.159389 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:11.159543 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:11.167382 (+ 7839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:11.167411 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:11.167435 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:11.167498 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:11.171837 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:11.173055 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:11.173066 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:11.173070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:11.173278 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:11.173824 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:11.173830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:11.173833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:11.174437 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:11.174763 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:11.175259 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 16:50:11.175532 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260812 16:50:11.177600 29319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:11.330004 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 29108
I20260812 16:50:11.359714 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 29346
Aug 12 16:50:11 dist-test-slave-1vmg krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:50:11.379492 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:11.360668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.28.243.3:41399)
0812 16:50:11.361180 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:11.361185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:11.361201 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:50:11.361268 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:11.361272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:11.361338 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:11.361437 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:11.364354 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:11.365251 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:11.366756 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:11.367401 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:11.367541 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:11.372786 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:11.372805 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:11.372823 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:11.372869 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:11.376344 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:11.377205 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:11.377212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:11.377215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:11.377380 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:11.377781 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:11.377785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:11.377787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:11.378222 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:11.378514 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:11.378865 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 16:50:11.379146 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":257,"threads_started":1}
I20260812 16:50:11.380342 29548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:11.531399 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 29346
I20260812 16:50:11.563077 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 28737
I20260812 16:50:11.565449 28830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:11.684194 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 28737
2026-08-12T16:50:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5421 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T16:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:11Z Disabled control of system clock
I20260812 16:50:11.792215 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46845
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:50:12.205444 29657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:12.205786 29657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:12.205896 29657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:12.214092 29657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:12.214252 29657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:12.214298 29657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:12.214398 29657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:12.214454 29657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:12.225759 29657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:12.227231 29657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:12.229178 29657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:12.243045 29672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:12.243077 29668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:12.243139 29669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:12.243896 29657 server_base.cc:1034] running on GCE node
I20260812 16:50:12.244766 29657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:12.246404 29657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:12.247799 29657 hybrid_clock.cc:648] HybridClock initialized: now 1786553412247603 us; error 43 us; skew 500 ppm
I20260812 16:50:12.248471 29657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:12.251789 29657 webserver.cc:460] Webserver started at http://127.28.243.62:44123/ using document root <none> and password file <none>
I20260812 16:50:12.252820 29657 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:12.252951 29657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:12.253386 29657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:12.257400 29657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "44a259ddba4747c896cc2cca85cad172"
format_stamp: "Formatted at 2026-08-12 16:50:12 on dist-test-slave-1vmg"
server_key: "00cd546966911a940b863ce89d91362c"
server_key_iv: "7d238563363f7166d91698aa67afc15b"
server_key_version: "encryptionkey@0"
I20260812 16:50:12.259048 29657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "44a259ddba4747c896cc2cca85cad172"
format_stamp: "Formatted at 2026-08-12 16:50:12 on dist-test-slave-1vmg"
server_key: "00cd546966911a940b863ce89d91362c"
server_key_iv: "7d238563363f7166d91698aa67afc15b"
server_key_version: "encryptionkey@0"
I20260812 16:50:12.266309 29657 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 16:50:12.270615 29683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:12.272626 29657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:50:12.273133 29657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "44a259ddba4747c896cc2cca85cad172"
format_stamp: "Formatted at 2026-08-12 16:50:12 on dist-test-slave-1vmg"
server_key: "00cd546966911a940b863ce89d91362c"
server_key_iv: "7d238563363f7166d91698aa67afc15b"
server_key_version: "encryptionkey@0"
I20260812 16:50:12.273643 29657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:12.292624 29657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:12.293581 29657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:12.293876 29657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:12.315869 29657 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:46845
I20260812 16:50:12.315891 29766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:46845 every 8 connection(s)
I20260812 16:50:12.317396 29657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:12.323412 29767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:12.323643 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 29657
I20260812 16:50:12.323869 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:12.324498 29644 external_mini_cluster.cc:1428] Setting key 2ae77e434cbb30be21ac16c2b7bb1c06
I20260812 16:50:12.339936 29767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172: Bootstrap starting.
I20260812 16:50:12.341688 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:12.326704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.28.243.62:46845)
0812 16:50:12.327872 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:12.327884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:12.327932 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:50:12.330311 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:12.330363 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:12.331458 (+ 1095us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:12.332279 (+ 821us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:12.333231 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:12.335189 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:12.336772 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:12.337603 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:12.337813 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:12.339072 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:12.339081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:12.339099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:12.339688 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:12.339789 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:12.339813 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:12.339928 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:12.340171 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 16:50:12.340514 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":415,"threads_started":1}
I20260812 16:50:12.343924 29767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:12.345230 29767 log.cc:824] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:12.348742 29767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172: No bootstrap required, opened a new log
I20260812 16:50:12.356691 29767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a259ddba4747c896cc2cca85cad172" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46845 } }
I20260812 16:50:12.357159 29767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:12.357270 29767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a259ddba4747c896cc2cca85cad172, State: Initialized, Role: FOLLOWER
I20260812 16:50:12.357924 29767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a259ddba4747c896cc2cca85cad172" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46845 } }
I20260812 16:50:12.358180 29767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:12.358326 29767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:12.358489 29767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:12.361111 29767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a259ddba4747c896cc2cca85cad172" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46845 } }
I20260812 16:50:12.361708 29767 leader_election.cc:302] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44a259ddba4747c896cc2cca85cad172; no voters:
I20260812 16:50:12.362393 29767 leader_election.cc:288] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:12.362563 29775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:12.363395 29775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [term 1 LEADER]: Becoming Leader. State: Replica: 44a259ddba4747c896cc2cca85cad172, State: Running, Role: LEADER
I20260812 16:50:12.363950 29775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a259ddba4747c896cc2cca85cad172" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46845 } }
I20260812 16:50:12.364745 29767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:12.366825 29776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44a259ddba4747c896cc2cca85cad172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a259ddba4747c896cc2cca85cad172" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46845 } } }
I20260812 16:50:12.367116 29777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a259ddba4747c896cc2cca85cad172. Latest consensus state: current_term: 1 leader_uuid: "44a259ddba4747c896cc2cca85cad172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a259ddba4747c896cc2cca85cad172" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46845 } } }
I20260812 16:50:12.367331 29776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:12.367395 29777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:12.375325 29786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:12.381937 29786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:12.394892 29786 catalog_manager.cc:1347] Generated new cluster ID: dae6a0009ead4f748689c71d726dc158
I20260812 16:50:12.395085 29786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:12.417611 29786 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:12.419684 29786 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:12.437202 29786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172: Generated new TSK 0
I20260812 16:50:12.438190 29786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:12.449435 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46845
--builtin_ntp_servers=127.28.243.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:50:12.908929 29810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:12.909273 29810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:12.909404 29810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:12.917676 29810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:12.917840 29810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:12.917986 29810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:50:12.927798 29810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:12.929240 29810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:12.932317 29810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:12.951860 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:12.951923 29822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:12.951903 29818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:12.953629 29810 server_base.cc:1034] running on GCE node
I20260812 16:50:12.954610 29810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:12.956632 29810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:12.957998 29810 hybrid_clock.cc:648] HybridClock initialized: now 1786553412957938 us; error 52 us; skew 500 ppm
I20260812 16:50:12.958462 29810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:12.961624 29810 webserver.cc:460] Webserver started at http://127.28.243.1:46019/ using document root <none> and password file <none>
I20260812 16:50:12.962536 29810 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:12.962663 29810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:12.963066 29810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:12.966547 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "a10024ca99624fb7978d90f1c7b19d6a"
format_stamp: "Formatted at 2026-08-12 16:50:12 on dist-test-slave-1vmg"
server_key: "cf0a85fc7f065db5344f4df7d3c22c23"
server_key_iv: "923b25ebb95bca5c6cf613fd792069f9"
server_key_version: "encryptionkey@0"
I20260812 16:50:12.968036 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "a10024ca99624fb7978d90f1c7b19d6a"
format_stamp: "Formatted at 2026-08-12 16:50:12 on dist-test-slave-1vmg"
server_key: "cf0a85fc7f065db5344f4df7d3c22c23"
server_key_iv: "923b25ebb95bca5c6cf613fd792069f9"
server_key_version: "encryptionkey@0"
I20260812 16:50:12.976156 29810 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 16:50:12.980865 29835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:12.982514 29810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 16:50:12.982715 29810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "a10024ca99624fb7978d90f1c7b19d6a"
format_stamp: "Formatted at 2026-08-12 16:50:12 on dist-test-slave-1vmg"
server_key: "cf0a85fc7f065db5344f4df7d3c22c23"
server_key_iv: "923b25ebb95bca5c6cf613fd792069f9"
server_key_version: "encryptionkey@0"
I20260812 16:50:12.983042 29810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:13.005587 29810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:13.006539 29810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:13.006863 29810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:13.008076 29810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:13.010025 29810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:13.010152 29810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:13.010483 29810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:13.010628 29810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:13.056582 29810 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:43071
I20260812 16:50:13.056551 30027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:43071 every 8 connection(s)
I20260812 16:50:13.058145 29810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:50:13.067269 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 29810
I20260812 16:50:13.067466 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:50:13.067909 29644 external_mini_cluster.cc:1428] Setting key e520afd6552c779f1e6567ddf9e80609
I20260812 16:50:13.071755 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46845
--builtin_ntp_servers=127.28.243.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:50:13.076143 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:13.062115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:59993 (local address 127.28.243.62:46845)
0812 16:50:13.062695 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:13.062699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:13.064177 (+ 1478us) server_negotiation.cc:407] Connection header received
0812 16:50:13.066231 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:13.066238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:13.066344 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:13.066449 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:13.067908 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.068746 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.071575 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.072199 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.072362 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:13.074552 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:13.074561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:13.074565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:13.074594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:13.074648 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:13.074665 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:13.075042 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:13.075305 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 16:50:13.075427 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":301,"threads_started":1}
I20260812 16:50:13.077113 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:13.062811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46845 (local address 127.28.243.1:59993)
0812 16:50:13.063969 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:13.064021 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:13.065870 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:13.066785 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:13.066795 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:13.066958 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:13.067700 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:13.067732 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.069115 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.069124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:13.071207 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:13.071233 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.072525 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.072532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:13.072796 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:13.073972 (+ 1176us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:13.073975 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:13.074028 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:13.074144 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:13.075047 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:13.075060 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:13.075068 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:50:13.075257 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 16:50:13.075740 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":350,"threads_started":1}
I20260812 16:50:13.079861 30028 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46845
I20260812 16:50:13.080385 30028 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:13.081650 30028 heartbeater.cc:511] Master 127.28.243.62:46845 requested a full tablet report, sending...
I20260812 16:50:13.084709 29703 ts_manager.cc:194] Registered new tserver with Master: a10024ca99624fb7978d90f1c7b19d6a (127.28.243.1:43071)
I20260812 16:50:13.086763 29703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.1:59993
W20260812 16:50:13.474259 30036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:13.474612 30036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:13.474736 30036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:13.484472 30036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:13.484639 30036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:13.484800 30036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:50:13.495975 30036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:13.498083 30036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:13.500871 30036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:13.517069 30048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:13.517158 30049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:13.517160 30051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:13.518319 30036 server_base.cc:1034] running on GCE node
I20260812 16:50:13.519081 30036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:13.520290 30036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:13.521562 30036 hybrid_clock.cc:648] HybridClock initialized: now 1786553413521508 us; error 43 us; skew 500 ppm
I20260812 16:50:13.521945 30036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:13.528475 30036 webserver.cc:460] Webserver started at http://127.28.243.2:37745/ using document root <none> and password file <none>
I20260812 16:50:13.529341 30036 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:13.529456 30036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:13.529807 30036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:13.534279 30036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "592562ae16224a40bae931fb6d11111a"
format_stamp: "Formatted at 2026-08-12 16:50:13 on dist-test-slave-1vmg"
server_key: "d02cda62e498cb5683f1890ac190b413"
server_key_iv: "d12e6eeecf07a217ee7b469d31260fa9"
server_key_version: "encryptionkey@0"
I20260812 16:50:13.535496 30036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "592562ae16224a40bae931fb6d11111a"
format_stamp: "Formatted at 2026-08-12 16:50:13 on dist-test-slave-1vmg"
server_key: "d02cda62e498cb5683f1890ac190b413"
server_key_iv: "d12e6eeecf07a217ee7b469d31260fa9"
server_key_version: "encryptionkey@0"
I20260812 16:50:13.542896 30036 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 16:50:13.548143 30057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:13.550559 30036 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:50:13.550979 30036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "592562ae16224a40bae931fb6d11111a"
format_stamp: "Formatted at 2026-08-12 16:50:13 on dist-test-slave-1vmg"
server_key: "d02cda62e498cb5683f1890ac190b413"
server_key_iv: "d12e6eeecf07a217ee7b469d31260fa9"
server_key_version: "encryptionkey@0"
I20260812 16:50:13.551445 30036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:13.569849 30036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:13.570845 30036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:13.571213 30036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:13.572425 30036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:13.574442 30036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:13.574549 30036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:13.574676 30036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:13.574755 30036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:13.607978 30036 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:46051
I20260812 16:50:13.608026 30231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:46051 every 8 connection(s)
I20260812 16:50:13.609392 30036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:50:13.615499 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30036
I20260812 16:50:13.615718 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:50:13.616780 29644 external_mini_cluster.cc:1428] Setting key fa06f048ceb2e17ca9dba320ebba9e39
I20260812 16:50:13.621783 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46845
--builtin_ntp_servers=127.28.243.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:50:13.627261 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:13.612422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:54331 (local address 127.28.243.62:46845)
0812 16:50:13.612924 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:13.612928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:13.614210 (+ 1282us) server_negotiation.cc:407] Connection header received
0812 16:50:13.617420 (+ 3210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:13.617425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:13.617515 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:13.617673 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:13.619577 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.620843 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.623590 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.624045 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.624200 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:13.626125 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:13.626138 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:13.626144 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:13.626186 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:13.626264 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:13.626290 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:13.626539 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:13.626774 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 16:50:13.626890 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":237,"threads_started":1}
I20260812 16:50:13.629858 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:13.613038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46845 (local address 127.28.243.2:54331)
0812 16:50:13.614020 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:13.614062 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:13.616809 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:13.618018 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:13.618030 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:13.618174 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:13.619304 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:13.619345 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.621150 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.621158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:13.623344 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:13.623368 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:13.624340 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:13.624347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:13.624673 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:13.625514 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:13.625517 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:13.625581 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:13.625714 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:13.626525 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:13.626538 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:13.626547 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:50:13.626751 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 16:50:13.627636 (+ 885us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":299,"threads_started":1}
I20260812 16:50:13.632028 30235 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46845
I20260812 16:50:13.632488 30235 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:13.634270 30235 heartbeater.cc:511] Master 127.28.243.62:46845 requested a full tablet report, sending...
I20260812 16:50:13.636926 29703 ts_manager.cc:194] Registered new tserver with Master: 592562ae16224a40bae931fb6d11111a (127.28.243.2:46051)
I20260812 16:50:13.638118 29703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.2:54331
W20260812 16:50:14.024173 30245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:14.024592 30245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:14.024745 30245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:14.032917 30245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:14.033077 30245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:14.033229 30245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:50:14.044365 30245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:14.045852 30245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:14.047969 30245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:14.062800 30255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:14.062718 30260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:14.062767 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:14.063858 30245 server_base.cc:1034] running on GCE node
I20260812 16:50:14.064664 30245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:14.065949 30245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:14.067566 30245 hybrid_clock.cc:648] HybridClock initialized: now 1786553414067455 us; error 56 us; skew 500 ppm
I20260812 16:50:14.068176 30245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:14.071854 30245 webserver.cc:460] Webserver started at http://127.28.243.3:39145/ using document root <none> and password file <none>
I20260812 16:50:14.072721 30245 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:14.072845 30245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:14.073342 30245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:14.077535 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "0a78cf3a2fcb4829b96a469e632c6818"
format_stamp: "Formatted at 2026-08-12 16:50:14 on dist-test-slave-1vmg"
server_key: "6c2b45dbc44c65cacd40edfa0891a287"
server_key_iv: "e86f6982e4c56b2e7683215fa91f4b61"
server_key_version: "encryptionkey@0"
I20260812 16:50:14.078665 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "0a78cf3a2fcb4829b96a469e632c6818"
format_stamp: "Formatted at 2026-08-12 16:50:14 on dist-test-slave-1vmg"
server_key: "6c2b45dbc44c65cacd40edfa0891a287"
server_key_iv: "e86f6982e4c56b2e7683215fa91f4b61"
server_key_version: "encryptionkey@0"
I20260812 16:50:14.085317 30245 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:50:14.090458 30274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:14.092593 30028 heartbeater.cc:503] Master 127.28.243.62:46845 was elected leader, sending a full tablet report...
I20260812 16:50:14.092794 30245 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:50:14.093025 30245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "0a78cf3a2fcb4829b96a469e632c6818"
format_stamp: "Formatted at 2026-08-12 16:50:14 on dist-test-slave-1vmg"
server_key: "6c2b45dbc44c65cacd40edfa0891a287"
server_key_iv: "e86f6982e4c56b2e7683215fa91f4b61"
server_key_version: "encryptionkey@0"
I20260812 16:50:14.093349 30245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:14.115994 30245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:14.116904 30245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:14.117219 30245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:14.118258 30245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:14.119901 30245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:14.120005 30245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:14.120122 30245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:14.120195 30245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:14.159081 30245 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:33691
I20260812 16:50:14.159134 30452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:33691 every 8 connection(s)
I20260812 16:50:14.160501 30245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:50:14.166759 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30245
I20260812 16:50:14.167155 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:50:14.167754 29644 external_mini_cluster.cc:1428] Setting key 46016ff1ee664fe0e76ac7d022bb88ad
I20260812 16:50:14.178908 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.163326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:38205 (local address 127.28.243.62:46845)
0812 16:50:14.163818 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.163822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.165190 (+ 1368us) server_negotiation.cc:407] Connection header received
0812 16:50:14.168667 (+ 3477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.168674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.168780 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.169039 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:14.172000 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.172826 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.174804 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.175354 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.175500 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.177619 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:14.177625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:14.177630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:14.177655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:14.177711 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:14.177731 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:14.177829 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.178263 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.178426 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260812 16:50:14.179837 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.163890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46845 (local address 127.28.243.3:38205)
0812 16:50:14.164981 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.165031 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.167949 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.170137 (+ 2188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.170157 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.170480 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:14.171671 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.171710 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.173057 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.173061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.174602 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.174622 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.175865 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.175886 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.176353 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.177156 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:14.177159 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:14.177211 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:14.177323 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:14.177858 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:14.177871 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:14.177880 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.178082 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.178523 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":304,"threads_started":1}
I20260812 16:50:14.182366 30454 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46845
I20260812 16:50:14.182878 30454 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:14.183992 30454 heartbeater.cc:511] Master 127.28.243.62:46845 requested a full tablet report, sending...
I20260812 16:50:14.185609 29703 ts_manager.cc:194] Registered new tserver with Master: 0a78cf3a2fcb4829b96a469e632c6818 (127.28.243.3:33691)
I20260812 16:50:14.186460 29703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.3:38205
I20260812 16:50:14.198192 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:50:14.211498 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.202904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.28.243.62:46845)
0812 16:50:14.203129 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.203134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.203613 (+ 479us) server_negotiation.cc:407] Connection header received
0812 16:50:14.203756 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.203761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.203832 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.203936 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:14.204837 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.205746 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.207817 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.208506 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.208669 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.210174 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:14.210181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:14.210185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:14.210217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:14.210271 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:14.210288 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:14.210469 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.210893 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.211156 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:50:14.216501 29703 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:50:14.223956 29703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:50:14.252362 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.239302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:46051 (local address 127.0.0.1:40900)
0812 16:50:14.241060 (+ 1758us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.241132 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.241360 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.242587 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.242597 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.242634 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.243229 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.243259 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.245509 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.245517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.248490 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.248523 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.251145 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.251154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.251441 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.251508 (+ 67us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.251680 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.251941 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"threads_started":1}
I20260812 16:50:14.254355 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.239523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.28.243.2:46051)
0812 16:50:14.240651 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.240662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.241612 (+ 950us) server_negotiation.cc:407] Connection header received
0812 16:50:14.241755 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.241764 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.242062 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.242272 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.243513 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.245289 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.248914 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.250681 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.250866 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.253653 (+ 2787us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.253766 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.253977 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":957,"threads_started":1}
I20260812 16:50:14.261493 30137 tablet_service.cc:1467] Processing CreateTablet for tablet ba57f971aa3b4534b3f5fda5e7c45406 (DEFAULT_TABLE table=test-table [id=231dbee3720348db98acec0c186d80af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:14.263396 30137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba57f971aa3b4534b3f5fda5e7c45406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:14.266038 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.241036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:33691 (local address 127.0.0.1:57128)
0812 16:50:14.253676 (+ 12640us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.253702 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.254982 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.257199 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.257208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.257267 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.257794 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.257814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.260325 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.260332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.262583 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.262604 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.264916 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.264926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.265170 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.265215 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.265374 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.265637 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12512,"mutex_wait_us":89,"thread_start_us":17133,"threads_started":1}
I20260812 16:50:14.266234 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.241713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.28.243.3:33691)
0812 16:50:14.247812 (+ 6099us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.247824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.253832 (+ 6008us) server_negotiation.cc:407] Connection header received
0812 16:50:14.255234 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.255247 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.255575 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.255818 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.258205 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.260089 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.262844 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.264643 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.264809 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.265244 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.265391 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.265776 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5882,"thread_start_us":5711,"threads_started":1}
I20260812 16:50:14.268006 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.239657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:42512)
0812 16:50:14.253015 (+ 13358us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.253044 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.253192 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.258643 (+ 5451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.258648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.258682 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.259003 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.259019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.261417 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.261424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.263626 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.263645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.267116 (+ 3471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.267124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.267353 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.267395 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.267501 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.267655 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13194,"mutex_wait_us":52,"thread_start_us":413,"threads_started":1}
I20260812 16:50:14.268205 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.240305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.28.243.1:43071)
0812 16:50:14.256463 (+ 16158us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.256474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.256515 (+ 41us) server_negotiation.cc:407] Connection header received
0812 16:50:14.256645 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.256654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.257191 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.259323 (+ 2132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.259655 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.261192 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.263989 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.266874 (+ 2885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.267071 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.267380 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.267647 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.267832 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15895,"thread_start_us":17529,"threads_started":1}
I20260812 16:50:14.270817 30341 tablet_service.cc:1467] Processing CreateTablet for tablet ba57f971aa3b4534b3f5fda5e7c45406 (DEFAULT_TABLE table=test-table [id=231dbee3720348db98acec0c186d80af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:14.272680 30341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba57f971aa3b4534b3f5fda5e7c45406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:14.273171 29919 tablet_service.cc:1467] Processing CreateTablet for tablet ba57f971aa3b4534b3f5fda5e7c45406 (DEFAULT_TABLE table=test-table [id=231dbee3720348db98acec0c186d80af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:14.274852 29919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba57f971aa3b4534b3f5fda5e7c45406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:14.279656 30486 tablet_bootstrap.cc:492] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Bootstrap starting.
I20260812 16:50:14.284006 30486 tablet_bootstrap.cc:654] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:14.285120 30486 log.cc:824] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:14.289362 30486 tablet_bootstrap.cc:492] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: No bootstrap required, opened a new log
I20260812 16:50:14.289732 30486 ts_tablet_manager.cc:1397] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260812 16:50:14.295579 30491 tablet_bootstrap.cc:492] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Bootstrap starting.
I20260812 16:50:14.295918 30492 tablet_bootstrap.cc:492] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Bootstrap starting.
I20260812 16:50:14.299544 30486 raft_consensus.cc:348] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.300315 30486 raft_consensus.cc:374] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:14.300410 30491 tablet_bootstrap.cc:654] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:14.300426 30486 raft_consensus.cc:729] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592562ae16224a40bae931fb6d11111a, State: Initialized, Role: FOLLOWER
I20260812 16:50:14.301234 30486 consensus_queue.cc:260] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.301721 30492 tablet_bootstrap.cc:654] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:14.301786 30491 log.cc:824] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:14.302641 30235 heartbeater.cc:503] Master 127.28.243.62:46845 was elected leader, sending a full tablet report...
I20260812 16:50:14.303402 30486 ts_tablet_manager.cc:1428] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260812 16:50:14.303946 30492 log.cc:824] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:14.307309 30491 tablet_bootstrap.cc:492] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: No bootstrap required, opened a new log
I20260812 16:50:14.307423 30492 tablet_bootstrap.cc:492] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: No bootstrap required, opened a new log
I20260812 16:50:14.307674 30491 ts_tablet_manager.cc:1397] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260812 16:50:14.307689 30492 ts_tablet_manager.cc:1397] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 16:50:14.316395 30491 raft_consensus.cc:348] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.316460 30492 raft_consensus.cc:348] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.316859 30491 raft_consensus.cc:374] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:14.316928 30492 raft_consensus.cc:374] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:14.316965 30491 raft_consensus.cc:729] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a78cf3a2fcb4829b96a469e632c6818, State: Initialized, Role: FOLLOWER
I20260812 16:50:14.317032 30492 raft_consensus.cc:729] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10024ca99624fb7978d90f1c7b19d6a, State: Initialized, Role: FOLLOWER
I20260812 16:50:14.317731 30491 consensus_queue.cc:260] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.317724 30492 consensus_queue.cc:260] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.319554 30454 heartbeater.cc:503] Master 127.28.243.62:46845 was elected leader, sending a full tablet report...
I20260812 16:50:14.319839 30492 ts_tablet_manager.cc:1428] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260812 16:50:14.321544 30491 ts_tablet_manager.cc:1428] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260812 16:50:14.362665 30236 tablet.cc:2369] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:14.409861 30493 raft_consensus.cc:482] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:14.410230 30493 raft_consensus.cc:504] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
W20260812 16:50:14.412313 30456 tablet.cc:2369] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:14.412447 30493 leader_election.cc:288] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10024ca99624fb7978d90f1c7b19d6a (127.28.243.1:43071), 0a78cf3a2fcb4829b96a469e632c6818 (127.28.243.3:33691)
I20260812 16:50:14.421451 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.413702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:33691 (local address 127.28.243.2:33845)
0812 16:50:14.414314 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.414328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.414438 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.414781 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.414785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.414810 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.415263 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.415276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.416517 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.416522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.418322 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.418342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.420274 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.420281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.420606 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.420729 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.420845 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.421057 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":141,"thread_start_us":270,"threads_started":1}
I20260812 16:50:14.421576 30498 raft_consensus.cc:482] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:14.421940 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.412877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.28.243.2:49439)
0812 16:50:14.413562 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.413579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.413756 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.414221 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.414226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.414257 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.414661 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.414675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.415878 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.415884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.418217 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.418256 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.420245 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.420258 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.420605 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.420726 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.420845 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.421058 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":67,"thread_start_us":409,"threads_started":1}
I20260812 16:50:14.421576 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.413985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:33845 (local address 127.28.243.3:33691)
0812 16:50:14.414174 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.414180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.414416 (+ 236us) server_negotiation.cc:407] Connection header received
0812 16:50:14.414564 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.414571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.414648 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.414761 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.415431 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.416344 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.418571 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.419894 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.420120 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.420200 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.420924 (+ 724us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.421231 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:50:14.422169 30498 raft_consensus.cc:504] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.422504 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.412927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:49439 (local address 127.28.243.1:43071)
0812 16:50:14.413154 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.413161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.413695 (+ 534us) server_negotiation.cc:407] Connection header received
0812 16:50:14.413904 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.413910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.414013 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.414111 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.414809 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.415737 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.418654 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.419953 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.420120 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.420200 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.420924 (+ 724us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.421231 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:50:14.423556 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba57f971aa3b4534b3f5fda5e7c45406" candidate_uuid: "592562ae16224a40bae931fb6d11111a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a78cf3a2fcb4829b96a469e632c6818" is_pre_election: true
I20260812 16:50:14.423556 29955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba57f971aa3b4534b3f5fda5e7c45406" candidate_uuid: "592562ae16224a40bae931fb6d11111a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10024ca99624fb7978d90f1c7b19d6a" is_pre_election: true
I20260812 16:50:14.424132 29955 raft_consensus.cc:2454] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592562ae16224a40bae931fb6d11111a in term 0.
I20260812 16:50:14.424091 30382 raft_consensus.cc:2454] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592562ae16224a40bae931fb6d11111a in term 0.
I20260812 16:50:14.424330 30498 leader_election.cc:288] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 592562ae16224a40bae931fb6d11111a (127.28.243.2:46051), a10024ca99624fb7978d90f1c7b19d6a (127.28.243.1:43071)
I20260812 16:50:14.425108 30061 leader_election.cc:302] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a78cf3a2fcb4829b96a469e632c6818, 592562ae16224a40bae931fb6d11111a; no voters:
I20260812 16:50:14.425508 30493 raft_consensus.cc:2781] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:14.425691 30493 raft_consensus.cc:482] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:14.425979 30493 raft_consensus.cc:3037] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:14.428752 30493 raft_consensus.cc:504] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.429500 30493 leader_election.cc:288] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [CANDIDATE]: Term 1 election: Requested vote from peers a10024ca99624fb7978d90f1c7b19d6a (127.28.243.1:43071), 0a78cf3a2fcb4829b96a469e632c6818 (127.28.243.3:33691)
I20260812 16:50:14.430119 29955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba57f971aa3b4534b3f5fda5e7c45406" candidate_uuid: "592562ae16224a40bae931fb6d11111a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10024ca99624fb7978d90f1c7b19d6a"
I20260812 16:50:14.430131 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba57f971aa3b4534b3f5fda5e7c45406" candidate_uuid: "592562ae16224a40bae931fb6d11111a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a78cf3a2fcb4829b96a469e632c6818"
I20260812 16:50:14.430356 29955 raft_consensus.cc:3037] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:14.430377 30382 raft_consensus.cc:3037] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:14.432086 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.425426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:57219 (local address 127.28.243.2:46051)
0812 16:50:14.425581 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.425586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.425602 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:50:14.425704 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.425710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.425787 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.425876 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.426453 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.427415 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.429603 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.430691 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.430857 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.431742 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.431797 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.431898 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 16:50:14.432534 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.424559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:46051 (local address 127.28.243.3:57219)
0812 16:50:14.424768 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.424786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.424926 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.426082 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.426087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.426117 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.426285 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.426293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.427613 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.427618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.429377 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.429394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.431035 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.431042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.431251 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.431288 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.431378 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.431537 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 16:50:14.433779 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.424775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.28.243.3:53733)
0812 16:50:14.426268 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.426296 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:14.426452 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:14.426775 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:14.426780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:14.426806 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:14.427205 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.427216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.428650 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.428655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.431047 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:14.431065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.433005 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.433012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:14.433201 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.433242 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 16:50:14.433334 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 16:50:14.433481 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":1774,"threads_started":1}
I20260812 16:50:14.433583 30167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba57f971aa3b4534b3f5fda5e7c45406" candidate_uuid: "0a78cf3a2fcb4829b96a469e632c6818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592562ae16224a40bae931fb6d11111a" is_pre_election: true
I20260812 16:50:14.433950 29955 raft_consensus.cc:2454] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592562ae16224a40bae931fb6d11111a in term 1.
I20260812 16:50:14.434156 30167 raft_consensus.cc:2379] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a78cf3a2fcb4829b96a469e632c6818 in current term 1: Already voted for candidate 592562ae16224a40bae931fb6d11111a in this term.
I20260812 16:50:14.434474 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.425227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:53733 (local address 127.28.243.1:43071)
0812 16:50:14.425418 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.425424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.426397 (+ 973us) server_negotiation.cc:407] Connection header received
0812 16:50:14.426582 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.426586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.426659 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.426746 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:14.427374 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.428473 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.431318 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.432634 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.432812 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.432882 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.434062 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.434216 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:50:14.434935 30061 leader_election.cc:302] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592562ae16224a40bae931fb6d11111a, a10024ca99624fb7978d90f1c7b19d6a; no voters:
I20260812 16:50:14.435264 30493 raft_consensus.cc:2781] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:14.435173 29954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba57f971aa3b4534b3f5fda5e7c45406" candidate_uuid: "0a78cf3a2fcb4829b96a469e632c6818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10024ca99624fb7978d90f1c7b19d6a" is_pre_election: true
I20260812 16:50:14.435551 29954 raft_consensus.cc:2379] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a78cf3a2fcb4829b96a469e632c6818 in current term 1: Already voted for candidate 592562ae16224a40bae931fb6d11111a in this term.
I20260812 16:50:14.435523 30382 raft_consensus.cc:2454] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592562ae16224a40bae931fb6d11111a in term 1.
I20260812 16:50:14.436028 30493 raft_consensus.cc:686] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 1 LEADER]: Becoming Leader. State: Replica: 592562ae16224a40bae931fb6d11111a, State: Running, Role: LEADER
I20260812 16:50:14.436343 30279 leader_election.cc:302] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a78cf3a2fcb4829b96a469e632c6818; no voters: 592562ae16224a40bae931fb6d11111a, a10024ca99624fb7978d90f1c7b19d6a
I20260812 16:50:14.436743 30498 raft_consensus.cc:2726] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 16:50:14.436617 30493 consensus_queue.cc:237] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } }
I20260812 16:50:14.442108 29703 catalog_manager.cc:5495] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a reported cstate change: term changed from 0 to 1, leader changed from <none> to 592562ae16224a40bae931fb6d11111a (127.28.243.2). New cstate: current_term: 1 leader_uuid: "592562ae16224a40bae931fb6d11111a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592562ae16224a40bae931fb6d11111a" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 46051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 } health_report { overall_health: UNKNOWN } } }
I20260812 16:50:14.470844 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.464762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.28.243.2:46051)
0812 16:50:14.464973 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.464980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.465086 (+ 106us) server_negotiation.cc:407] Connection header received
0812 16:50:14.465331 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.465335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.465447 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.465592 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:50:14.466062 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.466868 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.468463 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.468904 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.469023 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.469663 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:50:14.470185 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:50:14.470366 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.470512 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.470562 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:50:14.479877 30382 raft_consensus.cc:1261] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 1 FOLLOWER]: Refusing update from remote peer 592562ae16224a40bae931fb6d11111a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:14.479887 29954 raft_consensus.cc:1261] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 1 FOLLOWER]: Refusing update from remote peer 592562ae16224a40bae931fb6d11111a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:14.480871 30505 consensus_queue.cc:1035] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a78cf3a2fcb4829b96a469e632c6818" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:14.481668 30493 consensus_queue.cc:1035] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10024ca99624fb7978d90f1c7b19d6a" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:14.496479 30514 mvcc.cc:205] Tried to move back new op lower bound from 7317722785698926592 to 7317722785537191936. Current Snapshot: MvccSnapshot[applied={T|T < 7317722785698926592}]
I20260812 16:50:14.515034 29701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:14.515381 29701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:14.519740 29701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44a259ddba4747c896cc2cca85cad172: Sending DeleteTablet for 3 replicas of tablet ba57f971aa3b4534b3f5fda5e7c45406
I20260812 16:50:14.521052 30137 tablet_service.cc:1514] Processing DeleteTablet for tablet ba57f971aa3b4534b3f5fda5e7c45406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:14 UTC) from {username='slave'} at 127.0.0.1:40900
I20260812 16:50:14.521240 29919 tablet_service.cc:1514] Processing DeleteTablet for tablet ba57f971aa3b4534b3f5fda5e7c45406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:14 UTC) from {username='slave'} at 127.0.0.1:42512
I20260812 16:50:14.522130 30526 tablet_replica.cc:330] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: stopping tablet replica
I20260812 16:50:14.522240 30341 tablet_service.cc:1514] Processing DeleteTablet for tablet ba57f971aa3b4534b3f5fda5e7c45406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:14 UTC) from {username='slave'} at 127.0.0.1:57128
I20260812 16:50:14.522706 30526 raft_consensus.cc:2229] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:50:14.523545 30526 raft_consensus.cc:2258] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:14.526074 30526 ts_tablet_manager.cc:1905] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:14.526552 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 29810
I20260812 16:50:14.533133 30528 tablet_replica.cc:330] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: stopping tablet replica
I20260812 16:50:14.533983 30528 raft_consensus.cc:2229] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:14.534530 30526 ts_tablet_manager.cc:1918] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:14.534746 30528 raft_consensus.cc:2258] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:14.534837 30526 log.cc:1196] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/ba57f971aa3b4534b3f5fda5e7c45406
I20260812 16:50:14.535658 30526 ts_tablet_manager.cc:1939] T ba57f971aa3b4534b3f5fda5e7c45406 P 592562ae16224a40bae931fb6d11111a: Deleting consensus metadata
I20260812 16:50:14.538148 30528 ts_tablet_manager.cc:1905] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:14.539125 29686 catalog_manager.cc:4841] TS 592562ae16224a40bae931fb6d11111a (127.28.243.2:46051): tablet ba57f971aa3b4534b3f5fda5e7c45406 (table test-table [id=231dbee3720348db98acec0c186d80af]) successfully deleted
I20260812 16:50:14.544507 30527 tablet_replica.cc:330] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: stopping tablet replica
I20260812 16:50:14.545324 30527 raft_consensus.cc:2229] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:14.546326 30527 raft_consensus.cc:2258] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:14.546772 30528 ts_tablet_manager.cc:1918] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:14.547036 30528 log.cc:1196] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/ba57f971aa3b4534b3f5fda5e7c45406
I20260812 16:50:14.548370 30528 ts_tablet_manager.cc:1939] T ba57f971aa3b4534b3f5fda5e7c45406 P 0a78cf3a2fcb4829b96a469e632c6818: Deleting consensus metadata
I20260812 16:50:14.549829 30527 ts_tablet_manager.cc:1905] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:14.550590 29688 catalog_manager.cc:4841] TS 0a78cf3a2fcb4829b96a469e632c6818 (127.28.243.3:33691): tablet ba57f971aa3b4534b3f5fda5e7c45406 (table test-table [id=231dbee3720348db98acec0c186d80af]) successfully deleted
I20260812 16:50:14.554219 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.528384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.28.243.1:43071)
0812 16:50:14.528558 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.528565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.543934 (+ 15369us) server_negotiation.cc:407] Connection header received
0812 16:50:14.544553 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.544561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.544709 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.544819 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:14.546236 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.547693 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.549734 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.550352 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.550478 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.552287 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:14.552301 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:14.552321 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:14.553244 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:14.553319 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:14.553349 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:14.553501 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.553671 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.553809 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 16:50:14.555266 30023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:14.664482 30527 ts_tablet_manager.cc:1918] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:14.664768 30527 log.cc:1196] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/ba57f971aa3b4534b3f5fda5e7c45406
I20260812 16:50:14.665289 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 29810
I20260812 16:50:14.665462 30527 ts_tablet_manager.cc:1939] T ba57f971aa3b4534b3f5fda5e7c45406 P a10024ca99624fb7978d90f1c7b19d6a: Deleting consensus metadata
W20260812 16:50:14.692927 29688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:43071 (error 108)
I20260812 16:50:14.693171 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30036
I20260812 16:50:14.695245 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.694661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53112)
0812 16:50:14.694901 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.695059 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 16:50:14.695604 29688 catalog_manager.cc:4584] TS a10024ca99624fb7978d90f1c7b19d6a (127.28.243.1:43071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba57f971aa3b4534b3f5fda5e7c45406: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
I20260812 16:50:14.703919 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.694302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.28.243.2:46051)
0812 16:50:14.694463 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.694468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.694486 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:50:14.694549 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.694553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.694673 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.694774 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:14.695433 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.696648 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.698566 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.699691 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.699886 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.701554 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:14.701571 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:14.701599 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:14.702754 (+ 1155us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:14.702865 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:14.702899 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:14.703109 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.703326 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.703474 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:50:14.705019 30225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:14.749928 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.749281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53124)
0812 16:50:14.749588 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.749730 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 16:50:14.751212 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.750737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53140)
0812 16:50:14.750886 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.751015 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 16:50:14.812176 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.811587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53148)
0812 16:50:14.811861 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.811976 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 16:50:14.813458 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.813015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53152)
0812 16:50:14.813190 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.813305 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 16:50:14.833483 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30036
I20260812 16:50:14.867852 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30245
I20260812 16:50:14.874886 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.868690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.28.243.3:33691)
0812 16:50:14.868899 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:14.868906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:14.868927 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:50:14.869015 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:14.869018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:14.869103 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:14.869193 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:14.869598 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.870327 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.871677 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:14.872084 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:14.872204 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:14.873410 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:14.873420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:14.873438 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:14.874083 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:14.874138 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:14.874159 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:14.874270 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:14.874416 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 16:50:14.874538 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:50:14.875705 30447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:14.894052 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.893579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53160)
0812 16:50:14.893804 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.893895 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 16:50:14.895222 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:14.894735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53176)
0812 16:50:14.894970 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:14.895050 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 16:50:14.973006 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30245
I20260812 16:50:14.996408 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 29657
I20260812 16:50:14.997716 29759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:15.110414 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:15.109590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43071 (local address 127.0.0.1:53190)
0812 16:50:15.109929 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:15.110061 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260812 16:50:15.110595 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 29657
2026-08-12T16:50:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3430 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:15 dist-test-slave-1vmg krb5kdc[30548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:15 dist-test-slave-1vmg krb5kdc[30548](info): set up 2 sockets
Aug 12 16:50:15 dist-test-slave-1vmg krb5kdc[30548](info): commencing operation
krb5kdc: starting...
W20260812 16:50:16.501708 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.018s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:16 dist-test-slave-1vmg krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:50:16.884367 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:34107
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:34107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:17.307945 30570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:17.308290 30570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:17.308387 30570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:17.316223 30570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:17.316382 30570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:17.316442 30570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:17.316495 30570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:17.316548 30570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:17.328508 30570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:34107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:34107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:17.330076 30570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:17.331738 30570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:17.348242 30578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:17.348407 30577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:17.348506 30584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:17.349938 30570 server_base.cc:1034] running on GCE node
I20260812 16:50:17.351166 30570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:17.353039 30570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:17.354322 30570 hybrid_clock.cc:648] HybridClock initialized: now 1786553417354268 us; error 55 us; skew 500 ppm
I20260812 16:50:17.354740 30570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:17 dist-test-slave-1vmg krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553417, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:17.359221 30570 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:50:17.361083 30570 webserver.cc:460] Webserver started at http://127.28.243.62:42607/ using document root <none> and password file <none>
I20260812 16:50:17.361886 30570 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:17.362032 30570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:17.362380 30570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:17.365988 30570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "027560e7c309463a81317cba9ecf920f"
format_stamp: "Formatted at 2026-08-12 16:50:17 on dist-test-slave-1vmg"
server_key: "7f628c7b8dc294c4769f646a992df863"
server_key_iv: "05f4ed73a5389fefb51fc3fed4cf100f"
server_key_version: "encryptionkey@0"
I20260812 16:50:17.367164 30570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "027560e7c309463a81317cba9ecf920f"
format_stamp: "Formatted at 2026-08-12 16:50:17 on dist-test-slave-1vmg"
server_key: "7f628c7b8dc294c4769f646a992df863"
server_key_iv: "05f4ed73a5389fefb51fc3fed4cf100f"
server_key_version: "encryptionkey@0"
I20260812 16:50:17.374424 30570 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 16:50:17.379557 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:17.381327 30570 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:50:17.381573 30570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "027560e7c309463a81317cba9ecf920f"
format_stamp: "Formatted at 2026-08-12 16:50:17 on dist-test-slave-1vmg"
server_key: "7f628c7b8dc294c4769f646a992df863"
server_key_iv: "05f4ed73a5389fefb51fc3fed4cf100f"
server_key_version: "encryptionkey@0"
I20260812 16:50:17.381829 30570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:17.406877 30570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:17.413694 30570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:17.414377 30570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:17.437569 30671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:34107 every 8 connection(s)
I20260812 16:50:17.437572 30570 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:34107
I20260812 16:50:17.439465 30570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:17.445068 30672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:17.446292 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30570
I20260812 16:50:17.446485 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:17.447060 29644 external_mini_cluster.cc:1428] Setting key 5548a651a7e8beee5cb54e40b307d249
I20260812 16:50:17.462561 30672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f: Bootstrap starting.
Aug 12 16:50:17 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:17.467434 30672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:17.469076 30672 log.cc:824] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:17.474573 30672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f: No bootstrap required, opened a new log
I20260812 16:50:17.483515 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:17.450089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.28.243.62:34107)
0812 16:50:17.451553 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:17.451571 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:17.451641 (+ 70us) server_negotiation.cc:407] Connection header received
0812 16:50:17.453539 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:17.453571 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:17.454653 (+ 1082us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:17.455477 (+ 824us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:17.457921 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:17.459572 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:17.461593 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:17.462421 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:17.462588 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:17.468956 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:17.469036 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:17.469084 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:17.469163 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:17.475962 (+ 6799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:17.477434 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:17.477448 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:17.477462 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:17.477825 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:17.478411 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:17.478417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:17.478421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:17.479438 (+ 1017us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:17.479790 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:17.481107 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0812 16:50:17.481724 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":580,"threads_started":1}
I20260812 16:50:17.485252 30672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027560e7c309463a81317cba9ecf920f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34107 } }
I20260812 16:50:17.485715 30672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:17.486115 30672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027560e7c309463a81317cba9ecf920f, State: Initialized, Role: FOLLOWER
I20260812 16:50:17.486951 30672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027560e7c309463a81317cba9ecf920f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34107 } }
I20260812 16:50:17.488518 30672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:17.488703 30672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:17.488901 30672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:17.491386 30672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027560e7c309463a81317cba9ecf920f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34107 } }
I20260812 16:50:17.491964 30672 leader_election.cc:302] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 027560e7c309463a81317cba9ecf920f; no voters:
I20260812 16:50:17.492636 30672 leader_election.cc:288] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:17.492944 30677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:17.494026 30677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [term 1 LEADER]: Becoming Leader. State: Replica: 027560e7c309463a81317cba9ecf920f, State: Running, Role: LEADER
I20260812 16:50:17.494589 30677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027560e7c309463a81317cba9ecf920f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34107 } }
I20260812 16:50:17.495116 30672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:17.498452 30679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "027560e7c309463a81317cba9ecf920f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027560e7c309463a81317cba9ecf920f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34107 } } }
I20260812 16:50:17.498360 30681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 027560e7c309463a81317cba9ecf920f. Latest consensus state: current_term: 1 leader_uuid: "027560e7c309463a81317cba9ecf920f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027560e7c309463a81317cba9ecf920f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 34107 } } }
I20260812 16:50:17.498900 30679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:17.498960 30681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:17.499881 30693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:17.506956 30693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:17.521267 30693 catalog_manager.cc:1347] Generated new cluster ID: 6e4ecc7d3cf34bffb450940b2df317db
I20260812 16:50:17.521409 30693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:17.540380 30693 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:17.543111 30693 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:17.555701 30693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f: Generated new TSK 0
I20260812 16:50:17.557039 30693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
Entry for principal kudu/127.28.243.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab.
I20260812 16:50:17.656087 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34107
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:18.111986 30708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:18.112458 30708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:18.112882 30708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:18.122082 30708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:18.122264 30708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:18.122432 30708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:50:18.135617 30708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.1.keytab
--principal=kudu/127.28.243.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:18.137210 30708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:18.138703 30708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:18.155648 30716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:18.155694 30719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:18.155614 30715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:18.156631 30708 server_base.cc:1034] running on GCE node
I20260812 16:50:18.157370 30708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:18.158496 30708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:18.159747 30708 hybrid_clock.cc:648] HybridClock initialized: now 1786553418159691 us; error 41 us; skew 500 ppm
I20260812 16:50:18.160131 30708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:18 dist-test-slave-1vmg krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553418, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:18.163816 30708 init.cc:377] Logged in from keytab as kudu/127.28.243.1@KRBTEST.COM (short username kudu)
I20260812 16:50:18.165566 30708 webserver.cc:460] Webserver started at http://127.28.243.1:37307/ using document root <none> and password file <none>
I20260812 16:50:18.166354 30708 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:18.166493 30708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:18.166836 30708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:18.171234 30708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "09e05b3ff6a749b59ec7f1e58f8ab794"
format_stamp: "Formatted at 2026-08-12 16:50:18 on dist-test-slave-1vmg"
server_key: "e169c3b997c7f33c27896d742f6d6c9c"
server_key_iv: "74e3199dcdc97c1c967c53ed9887b984"
server_key_version: "encryptionkey@0"
I20260812 16:50:18.172281 30708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "09e05b3ff6a749b59ec7f1e58f8ab794"
format_stamp: "Formatted at 2026-08-12 16:50:18 on dist-test-slave-1vmg"
server_key: "e169c3b997c7f33c27896d742f6d6c9c"
server_key_iv: "74e3199dcdc97c1c967c53ed9887b984"
server_key_version: "encryptionkey@0"
I20260812 16:50:18.180186 30708 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 16:50:18.184716 30728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:18.186743 30708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:50:18.187019 30708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "09e05b3ff6a749b59ec7f1e58f8ab794"
format_stamp: "Formatted at 2026-08-12 16:50:18 on dist-test-slave-1vmg"
server_key: "e169c3b997c7f33c27896d742f6d6c9c"
server_key_iv: "74e3199dcdc97c1c967c53ed9887b984"
server_key_version: "encryptionkey@0"
I20260812 16:50:18.187289 30708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:18.235684 30708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:18.241851 30708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:18.242219 30708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:18.243362 30708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:18.244967 30708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:18.245064 30708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:18.245195 30708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:18.245251 30708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:18.282933 30708 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:43021
I20260812 16:50:18.283046 30920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:43021 every 8 connection(s)
I20260812 16:50:18.284341 30708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:50:18.293854 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30708
I20260812 16:50:18.294088 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:50:18.294828 29644 external_mini_cluster.cc:1428] Setting key cb43e993bdedd9160da3475e054746b6
Aug 12 16:50:18 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553418, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.1@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:18.312461 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:18.287158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:41343 (local address 127.28.243.62:34107)
0812 16:50:18.287624 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:18.287629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:18.289075 (+ 1446us) server_negotiation.cc:407] Connection header received
0812 16:50:18.291136 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:18.291142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:18.291237 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:18.291336 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:18.293223 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.294051 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.295962 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.296418 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.296560 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:18.303820 (+ 7260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:18.303828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:18.303832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:18.303905 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:18.307634 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:18.308697 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:18.308702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:18.308704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:18.308861 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:18.309358 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:18.309363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:18.309365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:18.309672 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:18.309826 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:18.311623 (+ 1797us) server_negotiation.cc:299] Negotiation successful
0812 16:50:18.312012 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260812 16:50:18.313874 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:18.287848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34107 (local address 127.28.243.1:41343)
0812 16:50:18.288895 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:18.288942 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:18.290776 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:18.291628 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:18.291638 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:18.292273 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:18.293016 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:18.293041 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.294233 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.294237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:18.295748 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:18.295765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.296794 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.296805 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:18.297333 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:18.299222 (+ 1889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:18.299260 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:18.303366 (+ 4106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:18.307898 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:18.307909 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:18.307926 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:18.308520 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:18.308987 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:18.308992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:18.308995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:18.309221 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:18.309818 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:18.309827 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:18.310653 (+ 826us) client_negotiation.cc:769] Sending connection context
0812 16:50:18.311381 (+ 728us) client_negotiation.cc:240] Negotiation successful
0812 16:50:18.312205 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":439,"threads_started":1}
I20260812 16:50:18.316260 30921 heartbeater.cc:347] Connected to a master server at 127.28.243.62:34107
I20260812 16:50:18.316664 30921 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:18.317541 30921 heartbeater.cc:511] Master 127.28.243.62:34107 requested a full tablet report, sending...
I20260812 16:50:18.320551 30616 ts_manager.cc:194] Registered new tserver with Master: 09e05b3ff6a749b59ec7f1e58f8ab794 (127.28.243.1:43021)
I20260812 16:50:18.322685 30616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.1@KRBTEST.COM'} at 127.28.243.1:41343
WARNING: no policy specified for kudu/127.28.243.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
Entry for principal kudu/127.28.243.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab.
I20260812 16:50:18.383483 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34107
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:18.775017 30931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:18.775377 30931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:18.775475 30931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:18.783512 30931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:18.783664 30931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:18.783790 30931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:50:18.794881 30931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.2.keytab
--principal=kudu/127.28.243.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:18.796440 30931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:18.798134 30931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:18.813037 30940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:18.813148 30942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:18.813930 30945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:18.814881 30931 server_base.cc:1034] running on GCE node
I20260812 16:50:18.815770 30931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:18.816938 30931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:18.818195 30931 hybrid_clock.cc:648] HybridClock initialized: now 1786553418818133 us; error 46 us; skew 500 ppm
I20260812 16:50:18.818573 30931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:18 dist-test-slave-1vmg krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553418, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:18.825313 30931 init.cc:377] Logged in from keytab as kudu/127.28.243.2@KRBTEST.COM (short username kudu)
I20260812 16:50:18.827221 30931 webserver.cc:460] Webserver started at http://127.28.243.2:37247/ using document root <none> and password file <none>
I20260812 16:50:18.828063 30931 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:18.828197 30931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:18.828522 30931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:18.831964 30931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8"
format_stamp: "Formatted at 2026-08-12 16:50:18 on dist-test-slave-1vmg"
server_key: "e0286ca4523178a192f813493428f010"
server_key_iv: "b473cbe478dd35e5b6aa4b8a6b64b00e"
server_key_version: "encryptionkey@0"
I20260812 16:50:18.832849 30931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8"
format_stamp: "Formatted at 2026-08-12 16:50:18 on dist-test-slave-1vmg"
server_key: "e0286ca4523178a192f813493428f010"
server_key_iv: "b473cbe478dd35e5b6aa4b8a6b64b00e"
server_key_version: "encryptionkey@0"
I20260812 16:50:18.840081 30931 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 16:50:18.844420 30957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:18.846401 30931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260812 16:50:18.846621 30931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8"
format_stamp: "Formatted at 2026-08-12 16:50:18 on dist-test-slave-1vmg"
server_key: "e0286ca4523178a192f813493428f010"
server_key_iv: "b473cbe478dd35e5b6aa4b8a6b64b00e"
server_key_version: "encryptionkey@0"
I20260812 16:50:18.846858 30931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:18.870489 30931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:18.877462 30931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:18.877826 30931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:18.879014 30931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:18.880712 30931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:18.880812 30931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:18.880949 30931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:18.881006 30931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:18.922811 30931 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:35311
I20260812 16:50:18.922892 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:35311 every 8 connection(s)
I20260812 16:50:18.924227 30931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:50:18.932488 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 30931
I20260812 16:50:18.932813 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:50:18.933319 29644 external_mini_cluster.cc:1428] Setting key ca02468e781b528bb8d239631e02da3a
Aug 12 16:50:18 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553418, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.2@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:18.952266 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:18.927104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:51409 (local address 127.28.243.62:34107)
0812 16:50:18.927672 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:18.927677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:18.929723 (+ 2046us) server_negotiation.cc:407] Connection header received
0812 16:50:18.932539 (+ 2816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:18.932547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:18.932666 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:18.932922 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:18.935694 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.936903 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.938818 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.939484 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.939642 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:18.943909 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:18.943915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:18.943918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:18.943964 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:18.947132 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:18.948609 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:18.948616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:18.948619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:18.949000 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:18.949704 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:18.949708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:18.949711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:18.950146 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:18.950303 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:18.951459 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0812 16:50:18.951770 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"spinlock_wait_cycles":20864,"thread_start_us":385,"threads_started":1}
I20260812 16:50:18.953339 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:18.928082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34107 (local address 127.28.243.2:51409)
0812 16:50:18.929490 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:18.929548 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:18.932045 (+ 2497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:18.933265 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:18.933278 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:18.934288 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:18.935396 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:18.935429 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.937175 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.937181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:18.938581 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:18.938612 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:18.939803 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:18.939809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:18.940059 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:18.940837 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:18.940864 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:18.943519 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:18.947542 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:18.947555 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:18.947585 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:18.948305 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:18.949189 (+ 884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:18.949194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:18.949197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:18.949506 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:18.950303 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:18.950313 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:18.950882 (+ 569us) client_negotiation.cc:769] Sending connection context
0812 16:50:18.951252 (+ 370us) client_negotiation.cc:240] Negotiation successful
0812 16:50:18.951908 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":476,"threads_started":1}
I20260812 16:50:18.955050 31143 heartbeater.cc:347] Connected to a master server at 127.28.243.62:34107
I20260812 16:50:18.955588 31143 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:18.956566 31143 heartbeater.cc:511] Master 127.28.243.62:34107 requested a full tablet report, sending...
I20260812 16:50:18.958992 30616 ts_manager.cc:194] Registered new tserver with Master: 7b22017f64984bcd8c9ef06e5ad8d3e8 (127.28.243.2:35311)
I20260812 16:50:18.960402 30616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.2@KRBTEST.COM'} at 127.28.243.2:51409
WARNING: no policy specified for kudu/127.28.243.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
Entry for principal kudu/127.28.243.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab.
I20260812 16:50:19.023409 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34107
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:50:19.327171 30921 heartbeater.cc:503] Master 127.28.243.62:34107 was elected leader, sending a full tablet report...
W20260812 16:50:19.448622 31154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:19.448954 31154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:19.449054 31154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:19.456843 31154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:19.457196 31154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:19.457456 31154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:50:19.468410 31154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.3.keytab
--principal=kudu/127.28.243.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:19.470098 31154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:19.471678 31154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:19.486640 31162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:19.486840 31164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:19.487210 31171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:19.488703 31154 server_base.cc:1034] running on GCE node
I20260812 16:50:19.489507 31154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:19.490612 31154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:19.494107 31154 hybrid_clock.cc:648] HybridClock initialized: now 1786553419494002 us; error 90 us; skew 500 ppm
I20260812 16:50:19.494545 31154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:19 dist-test-slave-1vmg krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553419, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:19.499161 31154 init.cc:377] Logged in from keytab as kudu/127.28.243.3@KRBTEST.COM (short username kudu)
I20260812 16:50:19.500995 31154 webserver.cc:460] Webserver started at http://127.28.243.3:33927/ using document root <none> and password file <none>
I20260812 16:50:19.501785 31154 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:19.501925 31154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:19.502269 31154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:19.505628 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "aa985646bc4d49dfae1d84b96312f8c4"
format_stamp: "Formatted at 2026-08-12 16:50:19 on dist-test-slave-1vmg"
server_key: "e9cb5247a1d6cf220d843da8ef7f347c"
server_key_iv: "da0f69e108b2f7fd12b9bca7296169a8"
server_key_version: "encryptionkey@0"
I20260812 16:50:19.506506 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "aa985646bc4d49dfae1d84b96312f8c4"
format_stamp: "Formatted at 2026-08-12 16:50:19 on dist-test-slave-1vmg"
server_key: "e9cb5247a1d6cf220d843da8ef7f347c"
server_key_iv: "da0f69e108b2f7fd12b9bca7296169a8"
server_key_version: "encryptionkey@0"
I20260812 16:50:19.513355 31154 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 16:50:19.518239 31189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:19.519969 31154 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:50:19.520179 31154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "aa985646bc4d49dfae1d84b96312f8c4"
format_stamp: "Formatted at 2026-08-12 16:50:19 on dist-test-slave-1vmg"
server_key: "e9cb5247a1d6cf220d843da8ef7f347c"
server_key_iv: "da0f69e108b2f7fd12b9bca7296169a8"
server_key_version: "encryptionkey@0"
I20260812 16:50:19.520416 31154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:19.541565 31154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:19.546625 31154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:19.547020 31154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:19.548323 31154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:19.549892 31154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:19.549980 31154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:19.550079 31154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:19.550130 31154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:19.589841 31154 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:33013
I20260812 16:50:19.589974 31383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:33013 every 8 connection(s)
I20260812 16:50:19.591629 31154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:50:19.596081 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31154
I20260812 16:50:19.596323 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:50:19.597213 29644 external_mini_cluster.cc:1428] Setting key c3e1786d8bfce50827ae1782c5551e56
Aug 12 16:50:19 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553419, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.3@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:19.618582 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.595484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:37337 (local address 127.28.243.62:34107)
0812 16:50:19.596157 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.596163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.600162 (+ 3999us) server_negotiation.cc:407] Connection header received
0812 16:50:19.600219 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.600223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.600302 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.600395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:19.602564 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.603428 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.606174 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.606629 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.606760 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.611056 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:19.611063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:19.611078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:19.611118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:19.614555 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:19.615487 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:19.615493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:19.615497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:19.615637 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:19.616106 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:19.616109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:19.616112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:19.616421 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:19.616584 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.617395 (+ 811us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.617666 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":321,"threads_started":1}
I20260812 16:50:19.619866 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.596149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:34107 (local address 127.28.243.3:37337)
0812 16:50:19.597344 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:19.597415 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:19.599702 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:19.600792 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:19.600806 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:19.601685 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:19.602336 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.602357 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.603734 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.603742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.605720 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.605748 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.607097 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.607110 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.607387 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.608133 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:50:19.608167 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:19.610718 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:19.614750 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:19.614760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:19.614779 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:19.615354 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:19.615755 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:50:19.615758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:50:19.615761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:50:19.615970 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:50:19.616557 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:19.616567 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:19.616981 (+ 414us) client_negotiation.cc:769] Sending connection context
0812 16:50:19.617255 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 16:50:19.617866 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":414,"threads_started":1}
I20260812 16:50:19.621744 31384 heartbeater.cc:347] Connected to a master server at 127.28.243.62:34107
I20260812 16:50:19.622102 31384 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:19.622983 31384 heartbeater.cc:511] Master 127.28.243.62:34107 requested a full tablet report, sending...
I20260812 16:50:19.625031 30616 ts_manager.cc:194] Registered new tserver with Master: aa985646bc4d49dfae1d84b96312f8c4 (127.28.243.3:33013)
I20260812 16:50:19.626194 30616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.3@KRBTEST.COM'} at 127.28.243.3:37337
I20260812 16:50:19.627816 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:50:19.648262 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.632631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.28.243.62:34107)
0812 16:50:19.632863 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.632870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.633157 (+ 287us) server_negotiation.cc:407] Connection header received
0812 16:50:19.633312 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.633316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.633384 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.633469 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:19.635829 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.636625 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.638227 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.638676 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.638784 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.641392 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:19.641397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:19.641400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:19.641443 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:19.644445 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:19.645316 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:19.645321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:19.645323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:19.645475 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:19.646017 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:19.646024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:19.646027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:19.646618 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:19.647076 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.647622 (+ 546us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.647886 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:50:19.653103 30616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:50:19.662084 30616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:50:19.695843 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.680842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:33013 (local address 127.0.0.1:55098)
0812 16:50:19.681968 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:19.682024 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:19.682218 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:19.686358 (+ 4140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:19.686367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:19.686414 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:19.686869 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.686886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.689272 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.689279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.692446 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.692467 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.694538 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.694547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.694784 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.694830 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 16:50:19.695109 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 16:50:19.695348 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":27,"thread_start_us":884,"threads_started":1}
I20260812 16:50:19.695827 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.681214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.28.243.3:33013)
0812 16:50:19.684786 (+ 3572us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.684796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.684835 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:50:19.684951 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.684959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.685356 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.685599 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:19.687142 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.689047 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.692853 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.694268 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.694424 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.694712 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.695155 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.695382 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3344,"thread_start_us":3129,"threads_started":1}
I20260812 16:50:19.696477 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.684151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:35311 (local address 127.0.0.1:46976)
0812 16:50:19.685377 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:19.685409 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:19.685650 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:19.687501 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:19.687507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:19.687543 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:19.688000 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.688012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.689831 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.689837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.692783 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.692805 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.695476 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.695485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.695677 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.695709 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:50:19.695774 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 16:50:19.696096 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":1,"thread_start_us":845,"threads_started":1}
I20260812 16:50:19.697875 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.684789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.28.243.2:35311)
0812 16:50:19.686378 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.686389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.686425 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:50:19.686537 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.686546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.686827 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.688244 (+ 1417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:19.688494 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.689664 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.693056 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.695204 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.697019 (+ 1815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.697270 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.697353 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.697545 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1381,"thread_start_us":1220,"threads_started":1}
I20260812 16:50:19.699802 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.681368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50228)
0812 16:50:19.683929 (+ 2561us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:19.683951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:19.684073 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:19.689844 (+ 5771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:19.689856 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:19.689902 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:19.690628 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.690645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.692654 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.692660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.695525 (+ 2865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.695538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.697874 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.697882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.698907 (+ 1025us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.699106 (+ 199us) client_negotiation.cc:769] Sending connection context
0812 16:50:19.699211 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 16:50:19.699391 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2431,"mutex_wait_us":4,"thread_start_us":2267,"threads_started":1}
I20260812 16:50:19.700289 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.682260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.28.243.1:43021)
0812 16:50:19.688821 (+ 6561us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.688835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.688883 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:50:19.689039 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.689049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.689518 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.689819 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:19.690866 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.692472 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.695747 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.697610 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.699216 (+ 1606us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.699484 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.699568 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.699771 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6165,"thread_start_us":6023,"threads_started":1}
I20260812 16:50:19.701038 31268 tablet_service.cc:1467] Processing CreateTablet for tablet d61a22c385a24fc19d38c5664269e409 (DEFAULT_TABLE table=test-table [id=1600b4975f294aeda846ea5d9be1756b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:19.702818 31268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61a22c385a24fc19d38c5664269e409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:19.702375 31036 tablet_service.cc:1467] Processing CreateTablet for tablet d61a22c385a24fc19d38c5664269e409 (DEFAULT_TABLE table=test-table [id=1600b4975f294aeda846ea5d9be1756b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:19.703195 30812 tablet_service.cc:1467] Processing CreateTablet for tablet d61a22c385a24fc19d38c5664269e409 (DEFAULT_TABLE table=test-table [id=1600b4975f294aeda846ea5d9be1756b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:19.704237 31036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61a22c385a24fc19d38c5664269e409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:19.704613 30812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61a22c385a24fc19d38c5664269e409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:19.717965 31411 tablet_bootstrap.cc:492] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Bootstrap starting.
I20260812 16:50:19.722061 31413 tablet_bootstrap.cc:492] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Bootstrap starting.
I20260812 16:50:19.723397 31411 tablet_bootstrap.cc:654] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:19.724938 31411 log.cc:824] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:19.726236 31413 tablet_bootstrap.cc:654] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:19.726667 31415 tablet_bootstrap.cc:492] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Bootstrap starting.
I20260812 16:50:19.728209 31413 log.cc:824] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:19.732225 31415 tablet_bootstrap.cc:654] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:19.732618 31411 tablet_bootstrap.cc:492] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: No bootstrap required, opened a new log
I20260812 16:50:19.732632 31413 tablet_bootstrap.cc:492] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: No bootstrap required, opened a new log
I20260812 16:50:19.732929 31413 ts_tablet_manager.cc:1397] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260812 16:50:19.732939 31411 ts_tablet_manager.cc:1397] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260812 16:50:19.733704 31415 log.cc:824] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:19.738046 31415 tablet_bootstrap.cc:492] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: No bootstrap required, opened a new log
I20260812 16:50:19.738410 31415 ts_tablet_manager.cc:1397] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260812 16:50:19.740970 31411 raft_consensus.cc:348] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.741401 31411 raft_consensus.cc:374] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:19.741500 31411 raft_consensus.cc:729] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e05b3ff6a749b59ec7f1e58f8ab794, State: Initialized, Role: FOLLOWER
I20260812 16:50:19.742141 31411 consensus_queue.cc:260] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.744148 31411 ts_tablet_manager.cc:1428] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260812 16:50:19.743875 31413 raft_consensus.cc:348] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.744339 31413 raft_consensus.cc:374] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:19.744436 31413 raft_consensus.cc:729] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa985646bc4d49dfae1d84b96312f8c4, State: Initialized, Role: FOLLOWER
I20260812 16:50:19.745097 31413 consensus_queue.cc:260] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.751861 31384 heartbeater.cc:503] Master 127.28.243.62:34107 was elected leader, sending a full tablet report...
I20260812 16:50:19.751398 31415 raft_consensus.cc:348] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.752766 31415 raft_consensus.cc:374] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:19.752903 31415 raft_consensus.cc:729] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b22017f64984bcd8c9ef06e5ad8d3e8, State: Initialized, Role: FOLLOWER
I20260812 16:50:19.753720 31413 ts_tablet_manager.cc:1428] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Time spent starting tablet: real 0.021s user 0.010s sys 0.011s
I20260812 16:50:19.753768 31415 consensus_queue.cc:260] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.755492 31143 heartbeater.cc:503] Master 127.28.243.62:34107 was elected leader, sending a full tablet report...
I20260812 16:50:19.756413 31415 ts_tablet_manager.cc:1428] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Time spent starting tablet: real 0.018s user 0.020s sys 0.000s
W20260812 16:50:19.787757 30922 tablet.cc:2369] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:19.799171 31425 raft_consensus.cc:482] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:19.799557 31425 raft_consensus.cc:504] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.801417 31425 leader_election.cc:288] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa985646bc4d49dfae1d84b96312f8c4 (127.28.243.3:33013), 09e05b3ff6a749b59ec7f1e58f8ab794 (127.28.243.1:43021)
I20260812 16:50:19.809403 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.802048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:42185 (local address 127.28.243.3:33013)
0812 16:50:19.802292 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.802300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.802694 (+ 394us) server_negotiation.cc:407] Connection header received
0812 16:50:19.803135 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.803143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.803250 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.803463 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:19.804240 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.804977 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.807240 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.808175 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.808344 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.808421 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.808957 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.809101 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:50:19.809607 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.801768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:33013 (local address 127.28.243.2:42185)
0812 16:50:19.802547 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:19.802574 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:19.802778 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:19.803521 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:19.803526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:19.803555 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:19.804069 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.804080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.805143 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.805147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.806946 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.807036 (+ 90us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.808567 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.808572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.808752 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.808782 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 16:50:19.808880 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 16:50:19.809041 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":64,"spinlock_wait_cycles":26496,"thread_start_us":530,"threads_started":1}
I20260812 16:50:19.810129 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.802273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:33751 (local address 127.28.243.1:43021)
0812 16:50:19.802487 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.802492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.803110 (+ 618us) server_negotiation.cc:407] Connection header received
0812 16:50:19.803284 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.803289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.803355 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.803437 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:19.804140 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.804895 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.807268 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.808530 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.808727 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.808819 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.809455 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.809759 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:50:19.810803 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.801831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.28.243.2:33751)
0812 16:50:19.803009 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:19.803024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:19.803142 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:19.803478 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:19.803483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:19.803509 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:19.803966 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.803981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.805044 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.805050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.806880 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:19.806898 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.808849 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.808858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:19.809065 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.809107 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:50:19.810247 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0812 16:50:19.810419 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":25,"thread_start_us":822,"threads_started":1}
I20260812 16:50:19.810804 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61a22c385a24fc19d38c5664269e409" candidate_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa985646bc4d49dfae1d84b96312f8c4" is_pre_election: true
I20260812 16:50:19.811502 31293 raft_consensus.cc:2454] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b22017f64984bcd8c9ef06e5ad8d3e8 in term 0.
I20260812 16:50:19.812125 30843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61a22c385a24fc19d38c5664269e409" candidate_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" is_pre_election: true
I20260812 16:50:19.812256 30962 leader_election.cc:302] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b22017f64984bcd8c9ef06e5ad8d3e8, aa985646bc4d49dfae1d84b96312f8c4; no voters:
I20260812 16:50:19.812660 31425 raft_consensus.cc:2781] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:19.812786 31425 raft_consensus.cc:482] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:19.812879 31425 raft_consensus.cc:3037] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:19.812858 30843 raft_consensus.cc:2454] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b22017f64984bcd8c9ef06e5ad8d3e8 in term 0.
I20260812 16:50:19.815560 31425 raft_consensus.cc:504] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.816273 31425 leader_election.cc:288] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [CANDIDATE]: Term 1 election: Requested vote from peers aa985646bc4d49dfae1d84b96312f8c4 (127.28.243.3:33013), 09e05b3ff6a749b59ec7f1e58f8ab794 (127.28.243.1:43021)
I20260812 16:50:19.816990 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61a22c385a24fc19d38c5664269e409" candidate_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa985646bc4d49dfae1d84b96312f8c4"
I20260812 16:50:19.817045 30843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61a22c385a24fc19d38c5664269e409" candidate_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794"
I20260812 16:50:19.817270 31293 raft_consensus.cc:3037] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:19.817313 30843 raft_consensus.cc:3037] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:19.819748 31293 raft_consensus.cc:2454] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b22017f64984bcd8c9ef06e5ad8d3e8 in term 1.
I20260812 16:50:19.819789 30843 raft_consensus.cc:2454] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b22017f64984bcd8c9ef06e5ad8d3e8 in term 1.
I20260812 16:50:19.820354 30962 leader_election.cc:302] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09e05b3ff6a749b59ec7f1e58f8ab794, 7b22017f64984bcd8c9ef06e5ad8d3e8, aa985646bc4d49dfae1d84b96312f8c4; no voters:
I20260812 16:50:19.820752 31425 raft_consensus.cc:2781] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:19.821266 31425 raft_consensus.cc:686] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7b22017f64984bcd8c9ef06e5ad8d3e8, State: Running, Role: LEADER
I20260812 16:50:19.821825 31425 consensus_queue.cc:237] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } }
I20260812 16:50:19.827186 30614 catalog_manager.cc:5495] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b22017f64984bcd8c9ef06e5ad8d3e8 (127.28.243.2). New cstate: current_term: 1 leader_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b22017f64984bcd8c9ef06e5ad8d3e8" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 35311 } health_report { overall_health: HEALTHY } } }
W20260812 16:50:19.844362 31385 tablet.cc:2369] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:19.852720 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.846290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.28.243.2:35311)
0812 16:50:19.846525 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.846532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.846553 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:50:19.846771 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.846775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.846876 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.847054 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:50:19.847436 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.848160 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.850079 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.850527 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.850669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.851474 (+ 805us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:50:19.852093 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:50:19.852232 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.852350 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.852439 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:50:19.861836 30843 raft_consensus.cc:1261] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 1 FOLLOWER]: Refusing update from remote peer 7b22017f64984bcd8c9ef06e5ad8d3e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:19.862306 31293 raft_consensus.cc:1261] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 1 FOLLOWER]: Refusing update from remote peer 7b22017f64984bcd8c9ef06e5ad8d3e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:19.863294 31428 consensus_queue.cc:1035] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa985646bc4d49dfae1d84b96312f8c4" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:19.863698 31425 consensus_queue.cc:1035] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e05b3ff6a749b59ec7f1e58f8ab794" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 43021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:19.895385 30616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:19.895742 30616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:19.900079 30616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 027560e7c309463a81317cba9ecf920f: Sending DeleteTablet for 3 replicas of tablet d61a22c385a24fc19d38c5664269e409
I20260812 16:50:19.901495 30812 tablet_service.cc:1514] Processing DeleteTablet for tablet d61a22c385a24fc19d38c5664269e409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:19 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:50228
I20260812 16:50:19.901499 31268 tablet_service.cc:1514] Processing DeleteTablet for tablet d61a22c385a24fc19d38c5664269e409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:19 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:55098
I20260812 16:50:19.902899 31036 tablet_service.cc:1514] Processing DeleteTablet for tablet d61a22c385a24fc19d38c5664269e409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:19 UTC) from {username='kudu', principal='kudu/127.28.243.62@KRBTEST.COM'} at 127.0.0.1:46976
I20260812 16:50:19.904732 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30708
I20260812 16:50:19.926249 31454 tablet_replica.cc:330] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: stopping tablet replica
I20260812 16:50:19.927131 31449 tablet_replica.cc:330] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: stopping tablet replica
I20260812 16:50:19.927621 31449 raft_consensus.cc:2229] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:19.928167 31454 raft_consensus.cc:2229] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:50:19.929723 31449 raft_consensus.cc:2258] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:19.930364 31452 tablet_replica.cc:330] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: stopping tablet replica
I20260812 16:50:19.930755 31452 raft_consensus.cc:2229] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:19.929025 31454 raft_consensus.cc:2258] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:19.934127 31454 ts_tablet_manager.cc:1905] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:19.936439 31449 ts_tablet_manager.cc:1905] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:19.937140 31452 raft_consensus.cc:2258] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 16:50:19 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.1@KRBTEST.COM
I20260812 16:50:19.943089 31452 ts_tablet_manager.cc:1905] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:19.943303 31454 ts_tablet_manager.cc:1918] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:19.943588 31454 log.cc:1196] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/d61a22c385a24fc19d38c5664269e409
I20260812 16:50:19.944319 31454 ts_tablet_manager.cc:1939] T d61a22c385a24fc19d38c5664269e409 P 7b22017f64984bcd8c9ef06e5ad8d3e8: Deleting consensus metadata
I20260812 16:50:19.946638 30596 catalog_manager.cc:4841] TS 7b22017f64984bcd8c9ef06e5ad8d3e8 (127.28.243.2:35311): tablet d61a22c385a24fc19d38c5664269e409 (table test-table [id=1600b4975f294aeda846ea5d9be1756b]) successfully deleted
I20260812 16:50:19.951916 31452 ts_tablet_manager.cc:1918] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:19.952332 31452 log.cc:1196] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/d61a22c385a24fc19d38c5664269e409
I20260812 16:50:19.953054 31452 ts_tablet_manager.cc:1939] T d61a22c385a24fc19d38c5664269e409 P aa985646bc4d49dfae1d84b96312f8c4: Deleting consensus metadata
I20260812 16:50:19.956741 30598 catalog_manager.cc:4841] TS aa985646bc4d49dfae1d84b96312f8c4 (127.28.243.3:33013): tablet d61a22c385a24fc19d38c5664269e409 (table test-table [id=1600b4975f294aeda846ea5d9be1756b]) successfully deleted
I20260812 16:50:19.961232 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:19.906317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.28.243.1:43021)
0812 16:50:19.906519 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:19.906526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:19.917164 (+ 10638us) server_negotiation.cc:407] Connection header received
0812 16:50:19.917748 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:19.917755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:19.917884 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:19.917966 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:19.920032 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.921252 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.928873 (+ 7621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:19.929616 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:19.929734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:19.948779 (+ 19045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:19.948805 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:19.948827 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:19.948883 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:19.957023 (+ 8140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:19.958426 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:19.958436 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:19.958439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:19.958709 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:19.959282 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:19.959286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:19.959288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:19.959772 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:19.960142 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:19.960513 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 16:50:19.960781 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:50:19.961978 30912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:20.107851 31449 ts_tablet_manager.cc:1918] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:20.108260 31449 log.cc:1196] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786553056142401-29644-0/minicluster-data/ts-0/wal/wals/d61a22c385a24fc19d38c5664269e409
I20260812 16:50:20.108923 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30708
I20260812 16:50:20.109040 31449 ts_tablet_manager.cc:1939] T d61a22c385a24fc19d38c5664269e409 P 09e05b3ff6a749b59ec7f1e58f8ab794: Deleting consensus metadata
W20260812 16:50:20.147449 30596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:43021 (error 108)
I20260812 16:50:20.147873 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30931
I20260812 16:50:20.149992 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.149532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50242)
0812 16:50:20.149704 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.149810 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 16:50:20.150336 30596 catalog_manager.cc:4584] TS 09e05b3ff6a749b59ec7f1e58f8ab794 (127.28.243.1:43021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d61a22c385a24fc19d38c5664269e409: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Aug 12 16:50:20 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.2@KRBTEST.COM
I20260812 16:50:20.169790 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.149090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.28.243.2:35311)
0812 16:50:20.149581 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:20.149587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:20.149615 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:50:20.149693 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:20.149698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:20.149808 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:20.149933 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:20.151983 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:20.153270 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:20.155344 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:20.156066 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:20.156264 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:20.161532 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:20.161553 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:20.161589 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:20.161662 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:20.165658 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:20.166636 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:20.166645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:20.166649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:20.166835 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:20.167473 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:20.167478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:20.167482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:20.167988 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:20.168323 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:20.169142 (+ 819us) server_negotiation.cc:299] Negotiation successful
0812 16:50:20.169396 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
I20260812 16:50:20.170737 31135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:20.204582 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.203987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50258)
0812 16:50:20.204257 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.204391 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 16:50:20.205753 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.205354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50268)
0812 16:50:20.205503 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.205593 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 16:50:20.266539 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.265909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50280)
0812 16:50:20.266182 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.266303 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 16:50:20.267808 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.267376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50288)
0812 16:50:20.267538 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.267630 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 16:50:20.302284 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30931
I20260812 16:50:20.335806 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31154
Aug 12 16:50:20 dist-test-slave-1vmg krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.3@KRBTEST.COM
I20260812 16:50:20.349486 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.348733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50304)
0812 16:50:20.349139 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.349268 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260812 16:50:20.350569 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.350198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50306)
0812 16:50:20.350386 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.350459 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 16:50:20.356650 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.337228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.28.243.3:33013)
0812 16:50:20.337941 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:20.337946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:20.337965 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:50:20.338029 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:20.338033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:20.338103 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:20.338219 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:20.339831 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:20.340605 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:20.342183 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:20.342801 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:20.343060 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:20.348541 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:20.348572 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:20.348604 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:20.348672 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:20.352655 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:20.353725 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:20.353737 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:20.353740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:20.353943 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:20.354421 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:20.354426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:20.354428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:20.354864 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:20.355342 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:20.355650 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 16:50:20.356164 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":464,"threads_started":1}
I20260812 16:50:20.357422 31377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:20.459255 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31154
I20260812 16:50:20.492589 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 30570
I20260812 16:50:20.494163 30667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:20.615594 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:20.614995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:43021 (local address 127.0.0.1:50312)
0812 16:50:20.615219 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:20.615327 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:43021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 16:50:20.615773 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 30570
2026-08-12T16:50:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5510 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T16:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:20Z Disabled control of system clock
I20260812 16:50:20.738623 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46473
--webserver_interface=127.28.243.62
--webserver_port=0
--builtin_ntp_servers=127.28.243.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:50:21.151580 31474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:21.151952 31474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:21.152051 31474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:21.160219 31474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:21.160374 31474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:21.160414 31474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:21.160467 31474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:21.160512 31474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:21.171823 31474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:21.173269 31474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:21.175374 31474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:21.188701 31482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:21.188956 31487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:21.188798 31483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:21.189873 31474 server_base.cc:1034] running on GCE node
I20260812 16:50:21.190750 31474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:21.192530 31474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:21.193769 31474 hybrid_clock.cc:648] HybridClock initialized: now 1786553421193722 us; error 96 us; skew 500 ppm
I20260812 16:50:21.194173 31474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:21.196774 31474 webserver.cc:460] Webserver started at http://127.28.243.62:33849/ using document root <none> and password file <none>
I20260812 16:50:21.197516 31474 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:21.197625 31474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:21.197965 31474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:21.201146 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "f42af77f8df64e8eb255c12a446d0836"
format_stamp: "Formatted at 2026-08-12 16:50:21 on dist-test-slave-1vmg"
server_key: "44b7e4d1ec6cdb432d95827f88424b0b"
server_key_iv: "455f757a010818f2efd92a30cbafde8e"
server_key_version: "encryptionkey@0"
I20260812 16:50:21.202013 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "f42af77f8df64e8eb255c12a446d0836"
format_stamp: "Formatted at 2026-08-12 16:50:21 on dist-test-slave-1vmg"
server_key: "44b7e4d1ec6cdb432d95827f88424b0b"
server_key_iv: "455f757a010818f2efd92a30cbafde8e"
server_key_version: "encryptionkey@0"
I20260812 16:50:21.208796 31474 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 16:50:21.213627 31496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:21.215823 31474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:50:21.216086 31474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "f42af77f8df64e8eb255c12a446d0836"
format_stamp: "Formatted at 2026-08-12 16:50:21 on dist-test-slave-1vmg"
server_key: "44b7e4d1ec6cdb432d95827f88424b0b"
server_key_iv: "455f757a010818f2efd92a30cbafde8e"
server_key_version: "encryptionkey@0"
I20260812 16:50:21.216342 31474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:21.242952 31474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:21.243938 31474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:21.244261 31474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:21.265679 31586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:46473 every 8 connection(s)
I20260812 16:50:21.265715 31474 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:46473
I20260812 16:50:21.267752 31474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:21.269767 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31474
I20260812 16:50:21.270018 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:21.270622 29644 external_mini_cluster.cc:1428] Setting key 6e9dcefbc646f16907bfa855a2686121
I20260812 16:50:21.273991 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:21.288434 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:21.273215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.28.243.62:46473)
0812 16:50:21.274479 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:21.274491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:21.274542 (+ 51us) server_negotiation.cc:407] Connection header received
0812 16:50:21.276483 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:21.276507 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:21.277051 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:21.277458 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:21.278147 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:21.279720 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:21.282080 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:21.283175 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:21.283405 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:21.285177 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:21.285188 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:21.285206 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:21.285891 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:21.286138 (+ 247us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:21.286185 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:21.286329 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:21.286687 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 16:50:21.287249 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":405,"threads_started":1}
I20260812 16:50:21.289606 31588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836: Bootstrap starting.
I20260812 16:50:21.294121 31588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:21.295385 31588 log.cc:824] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:21.299237 31588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836: No bootstrap required, opened a new log
I20260812 16:50:21.308209 31588 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42af77f8df64e8eb255c12a446d0836" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46473 } }
I20260812 16:50:21.308701 31588 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:21.308817 31588 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42af77f8df64e8eb255c12a446d0836, State: Initialized, Role: FOLLOWER
I20260812 16:50:21.309463 31588 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42af77f8df64e8eb255c12a446d0836" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46473 } }
I20260812 16:50:21.309732 31588 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:21.309882 31588 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:21.310048 31588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:21.314184 31588 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42af77f8df64e8eb255c12a446d0836" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46473 } }
I20260812 16:50:21.315140 31588 leader_election.cc:302] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42af77f8df64e8eb255c12a446d0836; no voters:
I20260812 16:50:21.316157 31588 leader_election.cc:288] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:21.316392 31596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:21.317318 31596 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [term 1 LEADER]: Becoming Leader. State: Replica: f42af77f8df64e8eb255c12a446d0836, State: Running, Role: LEADER
I20260812 16:50:21.318011 31596 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42af77f8df64e8eb255c12a446d0836" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46473 } }
I20260812 16:50:21.319017 31588 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:21.321414 31599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42af77f8df64e8eb255c12a446d0836. Latest consensus state: current_term: 1 leader_uuid: "f42af77f8df64e8eb255c12a446d0836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42af77f8df64e8eb255c12a446d0836" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46473 } } }
I20260812 16:50:21.321856 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42af77f8df64e8eb255c12a446d0836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42af77f8df64e8eb255c12a446d0836" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46473 } } }
I20260812 16:50:21.322227 31599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:21.322240 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:21.324949 31603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:21.341938 31603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:21.354102 31603 catalog_manager.cc:1347] Generated new cluster ID: bc5bb5947004490e8d9f7dac6e67b22c
I20260812 16:50:21.354424 31603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:21.370848 31603 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:21.372663 31603 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:21.384569 31603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836: Generated new TSK 0
I20260812 16:50:21.385561 31603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:21.403334 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.1:0
--local_ip_for_outbound_sockets=127.28.243.1
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46473
--builtin_ntp_servers=127.28.243.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:50:21.828936 31632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:21.829276 31632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:21.829391 31632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:21.837749 31632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:21.837895 31632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:21.838049 31632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.1
I20260812 16:50:21.848392 31632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.1
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.1
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:21.849857 31632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:21.851933 31632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:21.868005 31638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:21.867965 31641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:21.868028 31647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:21.868903 31632 server_base.cc:1034] running on GCE node
I20260812 16:50:21.869602 31632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:21.870711 31632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:21.871978 31632 hybrid_clock.cc:648] HybridClock initialized: now 1786553421871900 us; error 59 us; skew 500 ppm
I20260812 16:50:21.872354 31632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:21.875051 31632 webserver.cc:460] Webserver started at http://127.28.243.1:42279/ using document root <none> and password file <none>
I20260812 16:50:21.875864 31632 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:21.875984 31632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:21.876318 31632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:21.880506 31632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data/instance:
uuid: "7c8f043006f84361937e28006112eaf9"
format_stamp: "Formatted at 2026-08-12 16:50:21 on dist-test-slave-1vmg"
server_key: "0c5cdcc29ba5ccfba80fc8b82b364605"
server_key_iv: "886710f61d1199916899970ed8d77e37"
server_key_version: "encryptionkey@0"
I20260812 16:50:21.881505 31632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance:
uuid: "7c8f043006f84361937e28006112eaf9"
format_stamp: "Formatted at 2026-08-12 16:50:21 on dist-test-slave-1vmg"
server_key: "0c5cdcc29ba5ccfba80fc8b82b364605"
server_key_iv: "886710f61d1199916899970ed8d77e37"
server_key_version: "encryptionkey@0"
I20260812 16:50:21.888276 31632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 16:50:21.892680 31658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:21.894346 31632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:50:21.894582 31632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal
uuid: "7c8f043006f84361937e28006112eaf9"
format_stamp: "Formatted at 2026-08-12 16:50:21 on dist-test-slave-1vmg"
server_key: "0c5cdcc29ba5ccfba80fc8b82b364605"
server_key_iv: "886710f61d1199916899970ed8d77e37"
server_key_version: "encryptionkey@0"
I20260812 16:50:21.894833 31632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:21.918198 31632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:21.919276 31632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:21.919600 31632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:21.920768 31632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:21.923409 31632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:21.923566 31632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:21.923703 31632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:21.923786 31632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:21.961462 31632 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.1:41811
I20260812 16:50:21.961532 31864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.1:41811 every 8 connection(s)
I20260812 16:50:21.962994 31632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/data/info.pb
I20260812 16:50:21.967597 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31632
I20260812 16:50:21.967821 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-0/wal/instance
I20260812 16:50:21.968385 29644 external_mini_cluster.cc:1428] Setting key 2676f6e8b18fe6d18225e292011c6c2f
I20260812 16:50:21.971803 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.2:0
--local_ip_for_outbound_sockets=127.28.243.2
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46473
--builtin_ntp_servers=127.28.243.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:50:21.977943 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:21.966285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.1:42421 (local address 127.28.243.62:46473)
0812 16:50:21.967302 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:21.967309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:21.968256 (+ 947us) server_negotiation.cc:407] Connection header received
0812 16:50:21.970634 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:21.970639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:21.970711 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:21.970823 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:21.972165 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:21.972974 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:21.974799 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:21.975292 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:21.975414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:21.977007 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:21.977013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:21.977016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:21.977041 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:21.977106 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:21.977121 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:21.977205 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:21.977450 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 16:50:21.977572 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":512,"threads_started":1}
I20260812 16:50:21.979187 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:21.966764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46473 (local address 127.28.243.1:42421)
0812 16:50:21.968055 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:21.968115 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:21.970269 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:21.971127 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:21.971138 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:21.971247 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:21.971942 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:21.971972 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:21.973159 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:21.973163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:21.974619 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:21.974636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:21.975561 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:21.975569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:21.975818 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:21.976610 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:21.976613 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:21.976662 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:21.976765 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:21.977224 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:21.977236 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:21.977244 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:50:21.977427 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 16:50:21.977919 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":402,"threads_started":1}
I20260812 16:50:21.981078 31865 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46473
I20260812 16:50:21.981449 31865 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:21.982353 31865 heartbeater.cc:511] Master 127.28.243.62:46473 requested a full tablet report, sending...
I20260812 16:50:21.984810 31523 ts_manager.cc:194] Registered new tserver with Master: 7c8f043006f84361937e28006112eaf9 (127.28.243.1:41811)
I20260812 16:50:21.986470 31523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.1:42421
W20260812 16:50:22.394071 31870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:22.394398 31870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:22.394510 31870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:22.402720 31870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:22.402869 31870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:22.403041 31870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.2
I20260812 16:50:22.412994 31870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.2
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.2
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:22.414492 31870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:22.416569 31870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:22.434082 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:22.434234 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:22.434265 31881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:22.436450 31870 server_base.cc:1034] running on GCE node
I20260812 16:50:22.437327 31870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:22.438542 31870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:22.439811 31870 hybrid_clock.cc:648] HybridClock initialized: now 1786553422439754 us; error 58 us; skew 500 ppm
I20260812 16:50:22.440529 31870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:22.443590 31870 webserver.cc:460] Webserver started at http://127.28.243.2:39935/ using document root <none> and password file <none>
I20260812 16:50:22.444409 31870 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:22.444533 31870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:22.444886 31870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:22.448454 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data/instance:
uuid: "cac88b0f36124a5388f201cadd3f30a6"
format_stamp: "Formatted at 2026-08-12 16:50:22 on dist-test-slave-1vmg"
server_key: "560f7a8c0a0d84882192d7431c9ae8b4"
server_key_iv: "9c68211999345f9067047140962fc426"
server_key_version: "encryptionkey@0"
I20260812 16:50:22.449409 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance:
uuid: "cac88b0f36124a5388f201cadd3f30a6"
format_stamp: "Formatted at 2026-08-12 16:50:22 on dist-test-slave-1vmg"
server_key: "560f7a8c0a0d84882192d7431c9ae8b4"
server_key_iv: "9c68211999345f9067047140962fc426"
server_key_version: "encryptionkey@0"
I20260812 16:50:22.455773 31870 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:50:22.460829 31892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:22.462595 31870 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 16:50:22.462800 31870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal
uuid: "cac88b0f36124a5388f201cadd3f30a6"
format_stamp: "Formatted at 2026-08-12 16:50:22 on dist-test-slave-1vmg"
server_key: "560f7a8c0a0d84882192d7431c9ae8b4"
server_key_iv: "9c68211999345f9067047140962fc426"
server_key_version: "encryptionkey@0"
I20260812 16:50:22.463096 31870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:22.482801 31870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:22.483801 31870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:22.484138 31870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:22.485256 31870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:22.486944 31870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:22.487072 31870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 16:50:22.487191 31870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:22.487268 31870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:22.524802 31870 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.2:33213
I20260812 16:50:22.524859 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.2:33213 every 8 connection(s)
I20260812 16:50:22.526286 31870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/data/info.pb
I20260812 16:50:22.527743 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 31870
I20260812 16:50:22.528132 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal/instance
I20260812 16:50:22.528746 29644 external_mini_cluster.cc:1428] Setting key 7c2550a62027aea20bb8fd6936b0c29e
I20260812 16:50:22.533039 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.3:0
--local_ip_for_outbound_sockets=127.28.243.3
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46473
--builtin_ntp_servers=127.28.243.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:50:22.545604 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:22.530258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.2:53645 (local address 127.28.243.62:46473)
0812 16:50:22.531000 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:22.531007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:22.532729 (+ 1722us) server_negotiation.cc:407] Connection header received
0812 16:50:22.535398 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:22.535405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:22.535516 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:22.535661 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:22.537841 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:22.538884 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:22.541006 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:22.541603 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:22.541765 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:22.544428 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:22.544436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:22.544440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:22.544470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:22.544537 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:22.544556 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:22.544678 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:22.545111 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 16:50:22.545238 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":304,"threads_started":1}
I20260812 16:50:22.546674 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:22.531008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46473 (local address 127.28.243.2:53645)
0812 16:50:22.532491 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:22.532558 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:22.534984 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:22.535896 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:22.535906 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:22.536024 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:22.537459 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:22.537498 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:22.539214 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:22.539220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:22.540761 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:22.540784 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:22.542179 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:22.542193 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:22.542733 (+ 540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:22.543890 (+ 1157us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:22.543893 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:22.543960 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:22.544100 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:22.544792 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:22.544810 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:22.544821 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 16:50:22.545078 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 16:50:22.545530 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":483,"threads_started":1}
I20260812 16:50:22.548480 32104 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46473
I20260812 16:50:22.548820 32104 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:22.549705 32104 heartbeater.cc:511] Master 127.28.243.62:46473 requested a full tablet report, sending...
I20260812 16:50:22.551749 31523 ts_manager.cc:194] Registered new tserver with Master: cac88b0f36124a5388f201cadd3f30a6 (127.28.243.2:33213)
I20260812 16:50:22.552994 31523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.2:53645
W20260812 16:50:22.973418 32109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:22.973848 32109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:22.973999 32109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:22.984143 32109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:22.984287 32109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:22.984445 32109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.3
I20260812 16:50:22.990437 31865 heartbeater.cc:503] Master 127.28.243.62:46473 was elected leader, sending a full tablet report...
I20260812 16:50:22.997658 32109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.3
--webserver_port=0
--tserver_master_addrs=127.28.243.62:46473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.3
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:22.999402 32109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:23.001711 32109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:23.019955 32118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:23.019986 32120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:23.022285 32123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:23.025520 32109 server_base.cc:1034] running on GCE node
I20260812 16:50:23.027004 32109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:23.028949 32109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:23.032465 32109 hybrid_clock.cc:648] HybridClock initialized: now 1786553423032196 us; error 222 us; skew 500 ppm
I20260812 16:50:23.033063 32109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:50:23.036898 32109 webserver.cc:460] Webserver started at http://127.28.243.3:42983/ using document root <none> and password file <none>
I20260812 16:50:23.037949 32109 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:23.038091 32109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:23.038713 32109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:23.042397 32109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data/instance:
uuid: "ec74d7f186884c2f931a82098e9faa5b"
format_stamp: "Formatted at 2026-08-12 16:50:23 on dist-test-slave-1vmg"
server_key: "9d10e1ff295a9d02c52035684f3fce2f"
server_key_iv: "0fa7710c01461497bb4df4734e67c123"
server_key_version: "encryptionkey@0"
I20260812 16:50:23.043445 32109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance:
uuid: "ec74d7f186884c2f931a82098e9faa5b"
format_stamp: "Formatted at 2026-08-12 16:50:23 on dist-test-slave-1vmg"
server_key: "9d10e1ff295a9d02c52035684f3fce2f"
server_key_iv: "0fa7710c01461497bb4df4734e67c123"
server_key_version: "encryptionkey@0"
I20260812 16:50:23.051990 32109 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260812 16:50:23.057407 32134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:23.059553 32109 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260812 16:50:23.059806 32109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal
uuid: "ec74d7f186884c2f931a82098e9faa5b"
format_stamp: "Formatted at 2026-08-12 16:50:23 on dist-test-slave-1vmg"
server_key: "9d10e1ff295a9d02c52035684f3fce2f"
server_key_iv: "0fa7710c01461497bb4df4734e67c123"
server_key_version: "encryptionkey@0"
I20260812 16:50:23.060050 32109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:23.085920 32109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:23.087618 32109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:23.088029 32109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:23.089565 32109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:50:23.092244 32109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:50:23.092392 32109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:23.092532 32109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:50:23.092609 32109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:23.139183 32109 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.3:34429
I20260812 16:50:23.139183 32344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.3:34429 every 8 connection(s)
I20260812 16:50:23.140717 32109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/data/info.pb
I20260812 16:50:23.147754 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 32109
I20260812 16:50:23.147977 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal/instance
I20260812 16:50:23.148545 29644 external_mini_cluster.cc:1428] Setting key b73acbd50370b728ef0a1f426515e405
I20260812 16:50:23.159191 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.143856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:36301 (local address 127.28.243.62:46473)
0812 16:50:23.144353 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.144358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.146021 (+ 1663us) server_negotiation.cc:407] Connection header received
0812 16:50:23.148595 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.148815 (+ 220us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.148985 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.149139 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:23.151518 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.152794 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.155729 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.156386 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.156521 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.158145 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:23.158152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:23.158156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:23.158181 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:23.158237 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:23.158253 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:23.158337 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.158684 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.158798 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260812 16:50:23.160239 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.144593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46473 (local address 127.28.243.3:36301)
0812 16:50:23.145810 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.145864 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:23.147828 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:23.149597 (+ 1769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:23.149616 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:23.149806 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:50:23.151198 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.151258 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.153015 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.153023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.155456 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.155482 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.156680 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.156688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.156931 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.157705 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:50:23.157708 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:50:23.157755 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:50:23.157857 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:50:23.158389 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:50:23.158399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:50:23.158408 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:50:23.158597 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 16:50:23.159036 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":438,"threads_started":1}
I20260812 16:50:23.162024 32346 heartbeater.cc:347] Connected to a master server at 127.28.243.62:46473
I20260812 16:50:23.162392 32346 heartbeater.cc:464] Registering TS with master...
I20260812 16:50:23.163331 32346 heartbeater.cc:511] Master 127.28.243.62:46473 requested a full tablet report, sending...
I20260812 16:50:23.165156 31523 ts_manager.cc:194] Registered new tserver with Master: ec74d7f186884c2f931a82098e9faa5b (127.28.243.3:34429)
I20260812 16:50:23.166226 29644 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:50:23.166214 31523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.3:36301
I20260812 16:50:23.178983 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.171292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.28.243.62:46473)
0812 16:50:23.171709 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.171718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.172093 (+ 375us) server_negotiation.cc:407] Connection header received
0812 16:50:23.172285 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.172290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.172373 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.172596 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:23.173448 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.174289 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.176009 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.176493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.176616 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.178053 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:23.178072 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:23.178078 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:23.178111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:23.178172 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:23.178199 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:23.178300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.178480 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.178591 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:50:23.184965 31523 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:50:23.193286 31523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:50:23.230784 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.208882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.3:34429 (local address 127.0.0.1:50226)
0812 16:50:23.216946 (+ 8064us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.217031 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:23.217307 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:23.219243 (+ 1936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:23.219253 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:23.219297 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:23.220174 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.220206 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.223510 (+ 3304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.223520 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.226462 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.226498 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.229546 (+ 3048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.229555 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.229877 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.229946 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 16:50:23.230124 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 16:50:23.230384 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7683,"threads_started":1}
I20260812 16:50:23.231719 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.209139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.28.243.3:34429)
0812 16:50:23.218005 (+ 8866us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.218016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.218070 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:50:23.218204 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.218215 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.218513 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.218809 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:23.220699 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.223191 (+ 2492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.226929 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.228752 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.228951 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.229332 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.231109 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.231329 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8624,"thread_start_us":8454,"threads_started":1}
I20260812 16:50:23.245783 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.209697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:33213 (local address 127.0.0.1:60466)
0812 16:50:23.236052 (+ 26355us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.236084 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:23.236276 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:23.237524 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:23.237530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:23.237566 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:23.238018 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.238032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.240210 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.240217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.242821 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.242845 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.244893 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.244900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.245128 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.245164 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:50:23.245267 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 16:50:23.245424 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26192,"thread_start_us":31517,"threads_started":1}
I20260812 16:50:23.246622 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.210394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.28.243.2:33213)
0812 16:50:23.235295 (+ 24901us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.235306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.236433 (+ 1127us) server_negotiation.cc:407] Connection header received
0812 16:50:23.236576 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.236586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.236959 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.237196 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:23.238251 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.239995 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.243159 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.244288 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.244427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.244737 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.246052 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.246253 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24664,"thread_start_us":24925,"threads_started":1}
I20260812 16:50:23.249864 32229 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbda362cfde43e388cd930597ef56b5 (DEFAULT_TABLE table=test-table [id=c4e1fb90fac44df29af9a9b444e9f8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:23.251807 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.209288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39292)
0812 16:50:23.232338 (+ 23050us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.232362 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:23.232507 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:23.244414 (+ 11907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:23.244419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:23.244458 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:23.244736 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.244752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.246656 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.246662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.248685 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.248721 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.250784 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.250795 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.251066 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.251109 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 16:50:23.251189 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 16:50:23.251354 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22908,"thread_start_us":7105,"threads_started":1}
I20260812 16:50:23.251937 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbda362cfde43e388cd930597ef56b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:23.251410 31989 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbda362cfde43e388cd930597ef56b5 (DEFAULT_TABLE table=test-table [id=c4e1fb90fac44df29af9a9b444e9f8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:23.253047 31989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbda362cfde43e388cd930597ef56b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:23.253237 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.209957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.28.243.1:41811)
0812 16:50:23.243530 (+ 33573us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.243541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.243585 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:50:23.243734 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.243745 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.244105 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.244415 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:23.244947 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.246458 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.249020 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.250474 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.252280 (+ 1806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.252621 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.252717 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.252913 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33331,"thread_start_us":33158,"threads_started":1}
I20260812 16:50:23.257313 31748 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbda362cfde43e388cd930597ef56b5 (DEFAULT_TABLE table=test-table [id=c4e1fb90fac44df29af9a9b444e9f8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:50:23.259090 31748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbda362cfde43e388cd930597ef56b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:23.273226 32375 tablet_bootstrap.cc:492] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Bootstrap starting.
I20260812 16:50:23.281183 32375 tablet_bootstrap.cc:654] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:23.282908 32375 log.cc:824] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:23.284435 32373 tablet_bootstrap.cc:492] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Bootstrap starting.
I20260812 16:50:23.286731 32378 tablet_bootstrap.cc:492] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Bootstrap starting.
I20260812 16:50:23.288707 32373 tablet_bootstrap.cc:654] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:23.290192 32373 log.cc:824] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:23.290809 32378 tablet_bootstrap.cc:654] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:23.291000 32375 tablet_bootstrap.cc:492] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: No bootstrap required, opened a new log
I20260812 16:50:23.291376 32375 ts_tablet_manager.cc:1397] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20260812 16:50:23.292088 32378 log.cc:824] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:23.294356 32373 tablet_bootstrap.cc:492] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: No bootstrap required, opened a new log
I20260812 16:50:23.294690 32373 ts_tablet_manager.cc:1397] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 16:50:23.296389 32378 tablet_bootstrap.cc:492] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: No bootstrap required, opened a new log
I20260812 16:50:23.296681 32378 ts_tablet_manager.cc:1397] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 16:50:23.303872 32375 raft_consensus.cc:348] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.304458 32375 raft_consensus.cc:374] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:23.304563 32375 raft_consensus.cc:729] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec74d7f186884c2f931a82098e9faa5b, State: Initialized, Role: FOLLOWER
I20260812 16:50:23.304787 32378 raft_consensus.cc:348] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.305233 32378 raft_consensus.cc:374] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:23.305344 32378 raft_consensus.cc:729] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8f043006f84361937e28006112eaf9, State: Initialized, Role: FOLLOWER
I20260812 16:50:23.305318 32375 consensus_queue.cc:260] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.305932 32378 consensus_queue.cc:260] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.306600 32373 raft_consensus.cc:348] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.307482 32373 raft_consensus.cc:374] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:23.307677 32373 raft_consensus.cc:729] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac88b0f36124a5388f201cadd3f30a6, State: Initialized, Role: FOLLOWER
I20260812 16:50:23.308584 32375 ts_tablet_manager.cc:1428] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260812 16:50:23.309012 32346 heartbeater.cc:503] Master 127.28.243.62:46473 was elected leader, sending a full tablet report...
I20260812 16:50:23.310496 32378 ts_tablet_manager.cc:1428] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260812 16:50:23.308696 32373 consensus_queue.cc:260] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.311671 32104 heartbeater.cc:503] Master 127.28.243.62:46473 was elected leader, sending a full tablet report...
I20260812 16:50:23.312340 32373 ts_tablet_manager.cc:1428] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Time spent starting tablet: real 0.018s user 0.017s sys 0.001s
I20260812 16:50:23.390511 32387 raft_consensus.cc:482] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:50:23.391065 32387 raft_consensus.cc:504] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
W20260812 16:50:23.392725 32347 tablet.cc:2369] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:23.393961 32387 leader_election.cc:288] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c8f043006f84361937e28006112eaf9 (127.28.243.1:41811), cac88b0f36124a5388f201cadd3f30a6 (127.28.243.2:33213)
I20260812 16:50:23.402225 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.394255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:41741 (local address 127.28.243.1:41811)
0812 16:50:23.394464 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.394470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.394768 (+ 298us) server_negotiation.cc:407] Connection header received
0812 16:50:23.395306 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.395312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.395427 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.395537 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:23.396009 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.397100 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.399801 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.401178 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.401365 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.401444 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.401828 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.401969 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:50:23.402431 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.394716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.3:48897 (local address 127.28.243.2:33213)
0812 16:50:23.395100 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.395107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.395527 (+ 420us) server_negotiation.cc:407] Connection header received
0812 16:50:23.395722 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.395726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.395813 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.395897 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:50:23.396457 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.397597 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.400119 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.401480 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.401655 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.401723 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.402037 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.402184 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:50:23.402439 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.394624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.2:33213 (local address 127.28.243.3:48897)
0812 16:50:23.395400 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.395417 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:23.395555 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:23.395943 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:23.395947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:23.395972 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:23.396273 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.396282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.397807 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.397813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.399917 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.399928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.401643 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.401649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.401822 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.401850 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 16:50:23.401984 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 16:50:23.402122 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":28,"thread_start_us":491,"threads_started":1}
I20260812 16:50:23.402257 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.394238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.28.243.3:41741)
0812 16:50:23.394636 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.394655 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:50:23.394840 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:50:23.395609 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:50:23.395623 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:50:23.395650 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:50:23.395830 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.395838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.397312 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.397317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.399573 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:50:23.399599 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.401370 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.401376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:50:23.401563 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.401600 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:50:23.401712 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 16:50:23.401878 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":1}
I20260812 16:50:23.403697 32025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbda362cfde43e388cd930597ef56b5" candidate_uuid: "ec74d7f186884c2f931a82098e9faa5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cac88b0f36124a5388f201cadd3f30a6" is_pre_election: true
I20260812 16:50:23.403838 31787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbda362cfde43e388cd930597ef56b5" candidate_uuid: "ec74d7f186884c2f931a82098e9faa5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8f043006f84361937e28006112eaf9" is_pre_election: true
I20260812 16:50:23.404402 31787 raft_consensus.cc:2454] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec74d7f186884c2f931a82098e9faa5b in term 0.
I20260812 16:50:23.404399 32025 raft_consensus.cc:2454] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec74d7f186884c2f931a82098e9faa5b in term 0.
I20260812 16:50:23.405220 32142 leader_election.cc:302] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cac88b0f36124a5388f201cadd3f30a6, ec74d7f186884c2f931a82098e9faa5b; no voters:
I20260812 16:50:23.405839 32387 raft_consensus.cc:2781] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:50:23.406075 32387 raft_consensus.cc:482] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:50:23.406211 32387 raft_consensus.cc:3037] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:23.408780 32387 raft_consensus.cc:504] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.409540 32387 leader_election.cc:288] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [CANDIDATE]: Term 1 election: Requested vote from peers 7c8f043006f84361937e28006112eaf9 (127.28.243.1:41811), cac88b0f36124a5388f201cadd3f30a6 (127.28.243.2:33213)
I20260812 16:50:23.410212 31787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbda362cfde43e388cd930597ef56b5" candidate_uuid: "ec74d7f186884c2f931a82098e9faa5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8f043006f84361937e28006112eaf9"
I20260812 16:50:23.410211 32025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbda362cfde43e388cd930597ef56b5" candidate_uuid: "ec74d7f186884c2f931a82098e9faa5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cac88b0f36124a5388f201cadd3f30a6"
I20260812 16:50:23.410432 32025 raft_consensus.cc:3037] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:23.410427 31787 raft_consensus.cc:3037] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:23.412921 31787 raft_consensus.cc:2454] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec74d7f186884c2f931a82098e9faa5b in term 1.
I20260812 16:50:23.413895 32025 raft_consensus.cc:2454] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec74d7f186884c2f931a82098e9faa5b in term 1.
I20260812 16:50:23.413892 32140 leader_election.cc:302] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8f043006f84361937e28006112eaf9, ec74d7f186884c2f931a82098e9faa5b; no voters:
I20260812 16:50:23.414378 32387 raft_consensus.cc:2781] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:23.415089 32387 raft_consensus.cc:686] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 1 LEADER]: Becoming Leader. State: Replica: ec74d7f186884c2f931a82098e9faa5b, State: Running, Role: LEADER
I20260812 16:50:23.415742 32387 consensus_queue.cc:237] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } }
I20260812 16:50:23.421660 31522 catalog_manager.cc:5495] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec74d7f186884c2f931a82098e9faa5b (127.28.243.3). New cstate: current_term: 1 leader_uuid: "ec74d7f186884c2f931a82098e9faa5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec74d7f186884c2f931a82098e9faa5b" member_type: VOTER last_known_addr { host: "127.28.243.3" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 } health_report { overall_health: UNKNOWN } } }
I20260812 16:50:23.455789 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.447835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.28.243.3:34429)
0812 16:50:23.448033 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.448037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.448124 (+ 87us) server_negotiation.cc:407] Connection header received
0812 16:50:23.448253 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.448257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.448358 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.448506 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:50:23.448917 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.449812 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.452443 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.453185 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.453483 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.454203 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:50:23.454817 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:50:23.455245 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.455321 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.455452 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 16:50:23.466149 31866 tablet.cc:2369] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:50:23.466878 32025 raft_consensus.cc:1261] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 1 FOLLOWER]: Refusing update from remote peer ec74d7f186884c2f931a82098e9faa5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:23.466956 31787 raft_consensus.cc:1261] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 1 FOLLOWER]: Refusing update from remote peer ec74d7f186884c2f931a82098e9faa5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:50:23.468115 32387 consensus_queue.cc:1035] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8f043006f84361937e28006112eaf9" member_type: VOTER last_known_addr { host: "127.28.243.1" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:23.468502 32393 consensus_queue.cc:1035] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cac88b0f36124a5388f201cadd3f30a6" member_type: VOTER last_known_addr { host: "127.28.243.2" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:50:23.486188 32400 mvcc.cc:205] Tried to move back new op lower bound from 7317722822507089920 to 7317722822315229184. Current Snapshot: MvccSnapshot[applied={T|T < 7317722822507089920}]
I20260812 16:50:23.502269 31523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:23.502624 31523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:50:23.507051 31523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f42af77f8df64e8eb255c12a446d0836: Sending DeleteTablet for 3 replicas of tablet 3fbda362cfde43e388cd930597ef56b5
I20260812 16:50:23.509774 31748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbda362cfde43e388cd930597ef56b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:23 UTC) from {username='slave'} at 127.0.0.1:39292
I20260812 16:50:23.509774 32229 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbda362cfde43e388cd930597ef56b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:23 UTC) from {username='slave'} at 127.0.0.1:50226
I20260812 16:50:23.510298 31989 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbda362cfde43e388cd930597ef56b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:50:23 UTC) from {username='slave'} at 127.0.0.1:60466
I20260812 16:50:23.513574 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31632
I20260812 16:50:23.526160 32415 tablet_replica.cc:330] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: stopping tablet replica
I20260812 16:50:23.526784 32415 raft_consensus.cc:2229] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:23.527525 32415 raft_consensus.cc:2258] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:23.529872 32418 tablet_replica.cc:330] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: stopping tablet replica
I20260812 16:50:23.530748 32418 raft_consensus.cc:2229] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:50:23.532630 32418 raft_consensus.cc:2258] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:23.533334 32420 tablet_replica.cc:330] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: stopping tablet replica
I20260812 16:50:23.534833 32420 raft_consensus.cc:2229] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:50:23.535848 32415 ts_tablet_manager.cc:1905] T 3fbda362cfde43e388cd930597ef56b5 P 7c8f043006f84361937e28006112eaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:23.536082 32418 ts_tablet_manager.cc:1905] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:23.538017 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.515652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.28.243.1:41811)
0812 16:50:23.515900 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.515907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.518306 (+ 2399us) server_negotiation.cc:407] Connection header received
0812 16:50:23.518535 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.518543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.518716 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.518846 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:23.524481 (+ 5635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.525891 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.529884 (+ 3993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.530544 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.531177 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.535290 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:23.535304 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:23.535323 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:23.536515 (+ 1192us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:23.536614 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:23.536662 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:23.536833 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.537362 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.537548 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 16:50:23.540319 31860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:23.540688 32420 raft_consensus.cc:2258] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:50:23.544142 32418 ts_tablet_manager.cc:1918] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:23.544384 32418 log.cc:1196] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-2/wal/wals/3fbda362cfde43e388cd930597ef56b5
I20260812 16:50:23.545028 32418 ts_tablet_manager.cc:1939] T 3fbda362cfde43e388cd930597ef56b5 P ec74d7f186884c2f931a82098e9faa5b: Deleting consensus metadata
I20260812 16:50:23.547465 31499 catalog_manager.cc:4841] TS ec74d7f186884c2f931a82098e9faa5b (127.28.243.3:34429): tablet 3fbda362cfde43e388cd930597ef56b5 (table test-table [id=c4e1fb90fac44df29af9a9b444e9f8c2]) successfully deleted
I20260812 16:50:23.549309 32420 ts_tablet_manager.cc:1905] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:50:23.560282 32420 ts_tablet_manager.cc:1918] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:50:23.560485 32420 log.cc:1196] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Deleting WAL directory at /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786553056142401-29644-0/minicluster-data/ts-1/wal/wals/3fbda362cfde43e388cd930597ef56b5
I20260812 16:50:23.560941 32420 ts_tablet_manager.cc:1939] T 3fbda362cfde43e388cd930597ef56b5 P cac88b0f36124a5388f201cadd3f30a6: Deleting consensus metadata
I20260812 16:50:23.562731 31503 catalog_manager.cc:4841] TS cac88b0f36124a5388f201cadd3f30a6 (127.28.243.2:33213): tablet 3fbda362cfde43e388cd930597ef56b5 (table test-table [id=c4e1fb90fac44df29af9a9b444e9f8c2]) successfully deleted
I20260812 16:50:23.707490 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31632
I20260812 16:50:23.739262 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31870
W20260812 16:50:23.739243 31498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.1:41811 (error 108)
I20260812 16:50:23.741554 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.740944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39320)
0812 16:50:23.741186 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.741328 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 16:50:23.741866 31498 catalog_manager.cc:4584] TS 7c8f043006f84361937e28006112eaf9 (127.28.243.1:41811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fbda362cfde43e388cd930597ef56b5: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
I20260812 16:50:23.747663 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.740595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.28.243.2:33213)
0812 16:50:23.740814 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.740820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.740840 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:50:23.740917 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.740922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.741047 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.741161 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:23.741588 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.742460 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.743808 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.744244 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.744362 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.745905 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:23.745920 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:23.746014 (+ 94us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:23.746714 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:23.746772 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:23.746797 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:23.746977 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.747172 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.747302 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:50:23.749188 32094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:23.796201 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.795246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39328)
0812 16:50:23.795727 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.795935 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260812 16:50:23.797823 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.797363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39332)
0812 16:50:23.797535 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.797641 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 16:50:23.855696 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31870
I20260812 16:50:23.861439 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.860765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39338)
0812 16:50:23.861122 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.861239 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260812 16:50:23.862803 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.862299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39340)
0812 16:50:23.862519 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.862630 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260812 16:50:23.890179 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32109
I20260812 16:50:23.904624 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.891284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.28.243.3:34429)
0812 16:50:23.891529 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:23.891535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:23.891720 (+ 185us) server_negotiation.cc:407] Connection header received
0812 16:50:23.891884 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:23.891888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:23.891990 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:23.892126 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:23.892892 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.894282 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.896828 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:23.897502 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:23.897676 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:23.902728 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:23.902745 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:23.902766 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:23.903672 (+ 906us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:23.903736 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:23.903762 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:23.903862 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:23.904137 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 16:50:23.904248 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 16:50:23.905486 32334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:23.943950 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.943423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39352)
0812 16:50:23.943682 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.943783 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 16:50:23.945113 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:23.944683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39356)
0812 16:50:23.944881 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:23.944958 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 16:50:24.043372 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 32109
I20260812 16:50:24.074892 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 31474
I20260812 16:50:24.076303 31579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:24.194864 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:24.194303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.1:41811 (local address 127.0.0.1:39360)
0812 16:50:24.194563 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 16:50:24.194670 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.1:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 16:50:24.195147 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 31474
2026-08-12T16:50:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3585 ms)
[----------] 6 tests from EncryptionPolicyTest (26841 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:24 dist-test-slave-1vmg krb5kdc[32434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:24 dist-test-slave-1vmg krb5kdc[32434](info): set up 2 sockets
Aug 12 16:50:24 dist-test-slave-1vmg krb5kdc[32434](info): commencing operation
krb5kdc: starting...
W20260812 16:50:25.643112 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.381s user 0.000s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:25 dist-test-slave-1vmg krb5kdc[32434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:26.043896 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:26.445281 32458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:26.445663 32458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:26.445806 32458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:26.453876 32458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:26.454064 32458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:26.454146 32458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:26.454232 32458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:26.454295 32458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:26.466522 32458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:26.468032 32458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:26.469430 32458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:26.484230 32468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:26.484149 32473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:26.484163 32469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:26.484558 32458 server_base.cc:1034] running on GCE node
I20260812 16:50:26.485790 32458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:26.487597 32458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:26.488973 32458 hybrid_clock.cc:648] HybridClock initialized: now 1786553426488820 us; error 104 us; skew 500 ppm
I20260812 16:50:26.489639 32458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:26 dist-test-slave-1vmg krb5kdc[32434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:26.494588 32458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:26.497128 32458 webserver.cc:460] Webserver started at http://127.0.0.1:40337/ using document root <none> and password file <none>
I20260812 16:50:26.498380 32458 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:26.498565 32458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:26.499001 32458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:26.502731 32458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "7d043cf1e1484bdeb342a1ba7b57bc94"
format_stamp: "Formatted at 2026-08-12 16:50:26 on dist-test-slave-1vmg"
server_key: "46d577d08356a69c5746ce71f9e612c6"
server_key_iv: "dbb564e92426bdab7c8b2fcf6a03463a"
server_key_version: "encryptionkey@0"
I20260812 16:50:26.503790 32458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "7d043cf1e1484bdeb342a1ba7b57bc94"
format_stamp: "Formatted at 2026-08-12 16:50:26 on dist-test-slave-1vmg"
server_key: "46d577d08356a69c5746ce71f9e612c6"
server_key_iv: "dbb564e92426bdab7c8b2fcf6a03463a"
server_key_version: "encryptionkey@0"
I20260812 16:50:26.510659 32458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 16:50:26.515334 32486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:26.516897 32458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:50:26.517109 32458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "7d043cf1e1484bdeb342a1ba7b57bc94"
format_stamp: "Formatted at 2026-08-12 16:50:26 on dist-test-slave-1vmg"
server_key: "46d577d08356a69c5746ce71f9e612c6"
server_key_iv: "dbb564e92426bdab7c8b2fcf6a03463a"
server_key_version: "encryptionkey@0"
I20260812 16:50:26.517328 32458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:26.531023 32458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:26.537653 32458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:26.538036 32458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:26.561715 32458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41131
I20260812 16:50:26.561782 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41131 every 8 connection(s)
I20260812 16:50:26.563413 32458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:26.566390 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 32458
I20260812 16:50:26.566625 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:26.567507 29644 external_mini_cluster.cc:1428] Setting key 6cff5dfaa97c8cb67d6ce45bd3cc38ec
I20260812 16:50:26.568686 32570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:26.587189 32570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94: Bootstrap starting.
Aug 12 16:50:26 dist-test-slave-1vmg krb5kdc[32434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:50:26.591230 32570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:26.592500 32570 log.cc:824] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:26.596047 32570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94: No bootstrap required, opened a new log
I20260812 16:50:26.599666 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:26.570368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34757 (local address 127.0.0.1:41131)
0812 16:50:26.571567 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:26.571589 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:26.571655 (+ 66us) server_negotiation.cc:407] Connection header received
0812 16:50:26.574248 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:26.574288 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:26.575006 (+ 718us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:26.575532 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:26.578123 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:26.579863 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:26.582072 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:26.583079 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:26.583271 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:26.588760 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:26.588774 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:26.588949 (+ 175us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:26.589086 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:26.593764 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:26.595101 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:26.595111 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:26.595120 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:26.595344 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:26.595882 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:26.595886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:26.595888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:26.596664 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:26.597080 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:26.597640 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 16:50:26.598117 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":570,"threads_started":1}
I20260812 16:50:26.605815 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:26.606220 32570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260812 16:50:26.606787 32570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:26.606971 32570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d043cf1e1484bdeb342a1ba7b57bc94, State: Initialized, Role: FOLLOWER
I20260812 16:50:26.607733 32570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260812 16:50:26.608064 32570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:26.608234 32570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:26.608445 32570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:26.612782 32570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260812 16:50:26.613621 32570 leader_election.cc:302] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d043cf1e1484bdeb342a1ba7b57bc94; no voters:
I20260812 16:50:26.614418 32570 leader_election.cc:288] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:26.614535 32585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:26.616551 32585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [term 1 LEADER]: Becoming Leader. State: Replica: 7d043cf1e1484bdeb342a1ba7b57bc94, State: Running, Role: LEADER
I20260812 16:50:26.617372 32570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:26.617524 32585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260812 16:50:26.621825 32590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } } }
I20260812 16:50:26.622320 32590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:26.621428 32591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d043cf1e1484bdeb342a1ba7b57bc94. Latest consensus state: current_term: 1 leader_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d043cf1e1484bdeb342a1ba7b57bc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } } }
I20260812 16:50:26.623428 32591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:26.627931 32599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:26.636884 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:26.612834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49551 (local address 127.0.0.1:41131)
0812 16:50:26.613074 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:26.613082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:26.613575 (+ 493us) server_negotiation.cc:407] Connection header received
0812 16:50:26.613941 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:26.613949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:26.614042 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:26.614185 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:26.616351 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:26.617815 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:26.620105 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:26.621187 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:26.621380 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:26.626255 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:26.626265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:26.626271 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:26.626334 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:26.631751 (+ 5417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:26.633161 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:26.633168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:26.633172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:26.633381 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:26.634042 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:26.634048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:26.634051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:26.634554 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:26.634867 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:26.636216 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0812 16:50:26.636459 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:50:26.639649 32599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:26.655639 32599 catalog_manager.cc:1347] Generated new cluster ID: caf1e32cf3d741a09820df22112decaa
I20260812 16:50:26.655929 32599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:26.683458 32599 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:26.684900 32599 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:26.695762 32599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d043cf1e1484bdeb342a1ba7b57bc94: Generated new TSK 0
I20260812 16:50:26.697305 32599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:26.705231 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32458
I20260812 16:50:26.707459 32561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:26.807552 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 32458
W20260812 16:50:26.835590 32430 connection.cc:535] client connection to 127.0.0.1:41131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41131): Connection reset by peer (error 104)
2026-08-12T16:50:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2594 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:26 dist-test-slave-1vmg krb5kdc[32623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:26 dist-test-slave-1vmg krb5kdc[32623](info): set up 2 sockets
Aug 12 16:50:26 dist-test-slave-1vmg krb5kdc[32623](info): commencing operation
krb5kdc: starting...
W20260812 16:50:28.169281 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.004s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:28 dist-test-slave-1vmg krb5kdc[32623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:28.575881 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:29.027448 32646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:29.027805 32646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:29.028976 32646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:29.042665 32646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:29.042836 32646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:29.042884 32646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:29.042972 32646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:29.043020 32646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:29.056838 32646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:29.059175 32646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:29.060750 32646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:29.077246 32657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:29.077399 32658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:29.077204 32665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:29.078523 32646 server_base.cc:1034] running on GCE node
I20260812 16:50:29.079630 32646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:29.081521 32646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:29.082804 32646 hybrid_clock.cc:648] HybridClock initialized: now 1786553429082751 us; error 51 us; skew 500 ppm
I20260812 16:50:29.083287 32646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:29 dist-test-slave-1vmg krb5kdc[32623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:29.094278 32646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:29.096652 32646 webserver.cc:460] Webserver started at http://127.0.0.1:38477/ using document root <none> and password file <none>
I20260812 16:50:29.097944 32646 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:29.098218 32646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:29.098609 32646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:29.102180 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "f254f596f1ca49708b78efd55db0e662"
format_stamp: "Formatted at 2026-08-12 16:50:29 on dist-test-slave-1vmg"
server_key: "70d19fa9972570e1a6b5f611139149f4"
server_key_iv: "868fb986f32491eb184845cff3f94721"
server_key_version: "encryptionkey@0"
I20260812 16:50:29.103102 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "f254f596f1ca49708b78efd55db0e662"
format_stamp: "Formatted at 2026-08-12 16:50:29 on dist-test-slave-1vmg"
server_key: "70d19fa9972570e1a6b5f611139149f4"
server_key_iv: "868fb986f32491eb184845cff3f94721"
server_key_version: "encryptionkey@0"
I20260812 16:50:29.110086 32646 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 16:50:29.116437 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:29.118649 32646 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:50:29.119042 32646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "f254f596f1ca49708b78efd55db0e662"
format_stamp: "Formatted at 2026-08-12 16:50:29 on dist-test-slave-1vmg"
server_key: "70d19fa9972570e1a6b5f611139149f4"
server_key_iv: "868fb986f32491eb184845cff3f94721"
server_key_version: "encryptionkey@0"
I20260812 16:50:29.119318 32646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:29.141536 32646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:29.146565 32646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:29.146894 32646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:29.172554 307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37269 every 8 connection(s)
I20260812 16:50:29.172586 32646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37269
I20260812 16:50:29.175076 32646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:29.180075 308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:29.183822 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 32646
I20260812 16:50:29.184048 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:29.184512 29644 external_mini_cluster.cc:1428] Setting key 5afbb583bd0f5acb8c9fdc3b39bb63de
I20260812 16:50:29.195413 308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662: Bootstrap starting.
I20260812 16:50:29.199987 308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:29.201431 308 log.cc:824] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:29.202463 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:29.186660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35189 (local address 127.0.0.1:37269)
0812 16:50:29.187687 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:29.187701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:29.187762 (+ 61us) server_negotiation.cc:407] Connection header received
0812 16:50:29.190351 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:29.190400 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:29.191251 (+ 851us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:29.192075 (+ 824us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:29.192876 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:29.194690 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:29.196296 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:29.197227 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:29.197481 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:29.199479 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:29.199491 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:29.199513 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:29.200252 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:29.200369 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:29.200413 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:29.200627 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:29.200975 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 16:50:29.201200 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":478,"threads_started":1}
I20260812 16:50:29.205137 308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662: No bootstrap required, opened a new log
I20260812 16:50:29.207417 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:29.217140 308 raft_consensus.cc:348] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254f596f1ca49708b78efd55db0e662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260812 16:50:29.217895 308 raft_consensus.cc:374] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:29.218207 308 raft_consensus.cc:729] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f254f596f1ca49708b78efd55db0e662, State: Initialized, Role: FOLLOWER
I20260812 16:50:29.219225 308 consensus_queue.cc:260] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254f596f1ca49708b78efd55db0e662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260812 16:50:29.219610 308 raft_consensus.cc:388] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:29.219792 308 raft_consensus.cc:482] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:29.219981 308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:29.223131 308 raft_consensus.cc:504] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254f596f1ca49708b78efd55db0e662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260812 16:50:29.223958 308 leader_election.cc:302] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f254f596f1ca49708b78efd55db0e662; no voters:
I20260812 16:50:29.224241 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:29.214955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49881 (local address 127.0.0.1:37269)
0812 16:50:29.215551 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:29.215559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:29.216149 (+ 590us) server_negotiation.cc:407] Connection header received
0812 16:50:29.216599 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:29.216605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:29.216712 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:29.216842 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:29.218230 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:29.219430 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:29.221169 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:29.221880 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:29.222043 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:29.223297 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:29.223304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:29.223309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:29.223337 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:29.223440 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:29.223463 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:29.223583 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:29.223807 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 16:50:29.223871 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260812 16:50:29.225131 308 leader_election.cc:288] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:29.225250 325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:29.226518 325 raft_consensus.cc:686] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [term 1 LEADER]: Becoming Leader. State: Replica: f254f596f1ca49708b78efd55db0e662, State: Running, Role: LEADER
I20260812 16:50:29.227227 325 consensus_queue.cc:237] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254f596f1ca49708b78efd55db0e662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260812 16:50:29.227509 308 sys_catalog.cc:564] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:29.231248 333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f254f596f1ca49708b78efd55db0e662. Latest consensus state: current_term: 1 leader_uuid: "f254f596f1ca49708b78efd55db0e662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254f596f1ca49708b78efd55db0e662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } } }
I20260812 16:50:29.231874 333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:29.231352 332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f254f596f1ca49708b78efd55db0e662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254f596f1ca49708b78efd55db0e662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } } }
I20260812 16:50:29.232321 332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:29.242525 345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:29.248915 345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:29.261291 345 catalog_manager.cc:1347] Generated new cluster ID: 1caf2f36f0bd4b01bb626a1dae7923fa
I20260812 16:50:29.261445 345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:29.282951 345 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:29.285116 345 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:29.297712 345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f254f596f1ca49708b78efd55db0e662: Generated new TSK 0
I20260812 16:50:29.298988 345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:29.319435 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32646
W20260812 16:50:29.321059 300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35189
W20260812 16:50:29.322011 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 32646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:29.322213 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 32646
W20260812 16:50:29.364048 32620 connection.cc:535] client connection to 127.0.0.1:37269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37269): Connection reset by peer (error 104)
2026-08-12T16:50:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:29 dist-test-slave-1vmg krb5kdc[369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:29 dist-test-slave-1vmg krb5kdc[369](info): set up 2 sockets
Aug 12 16:50:29 dist-test-slave-1vmg krb5kdc[369](info): commencing operation
krb5kdc: starting...
W20260812 16:50:30.697054 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.312s user 0.000s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:30 dist-test-slave-1vmg krb5kdc[369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:31.093726 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:31.490615 393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:31.491007 393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:31.491119 393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:31.500283 393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:31.500439 393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:31.500486 393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:31.500551 393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:31.500599 393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:31.512203 393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:31.514380 393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:31.515921 393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:31.530112 401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:31.530159 409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:31.530444 403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:31.531322 393 server_base.cc:1034] running on GCE node
I20260812 16:50:31.532610 393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:31.535149 393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:31.536455 393 hybrid_clock.cc:648] HybridClock initialized: now 1786553431536398 us; error 51 us; skew 500 ppm
I20260812 16:50:31.536890 393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:31 dist-test-slave-1vmg krb5kdc[369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:31.545655 393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:31.547773 393 webserver.cc:460] Webserver started at http://127.0.0.1:42461/ using document root <none> and password file <none>
I20260812 16:50:31.548626 393 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:31.548774 393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:31.549122 393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:31.554297 393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "e4400f315ca14aecaa06a40cc87487b0"
format_stamp: "Formatted at 2026-08-12 16:50:31 on dist-test-slave-1vmg"
server_key: "f0ea5547e7ff5b8316d4e1654db17525"
server_key_iv: "901ea6a233f3ccfdf08294996a1c60e8"
server_key_version: "encryptionkey@0"
I20260812 16:50:31.555702 393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "e4400f315ca14aecaa06a40cc87487b0"
format_stamp: "Formatted at 2026-08-12 16:50:31 on dist-test-slave-1vmg"
server_key: "f0ea5547e7ff5b8316d4e1654db17525"
server_key_iv: "901ea6a233f3ccfdf08294996a1c60e8"
server_key_version: "encryptionkey@0"
I20260812 16:50:31.565716 393 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.003s sys 0.006s
I20260812 16:50:31.576431 420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:31.579255 393 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.005s
I20260812 16:50:31.579511 393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "e4400f315ca14aecaa06a40cc87487b0"
format_stamp: "Formatted at 2026-08-12 16:50:31 on dist-test-slave-1vmg"
server_key: "f0ea5547e7ff5b8316d4e1654db17525"
server_key_iv: "901ea6a233f3ccfdf08294996a1c60e8"
server_key_version: "encryptionkey@0"
I20260812 16:50:31.579821 393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:31.591404 393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:31.597554 393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:31.597909 393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:31.621394 393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34465
I20260812 16:50:31.621479 507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34465 every 8 connection(s)
I20260812 16:50:31.623931 393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:31.627537 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 393
I20260812 16:50:31.627789 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:31.628350 29644 external_mini_cluster.cc:1428] Setting key dac07f6dcdd571a93cfecb4f679b5f0f
I20260812 16:50:31.629235 508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:31.640476 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:31.630821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53589 (local address 127.0.0.1:34465)
0812 16:50:31.632007 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:31.632027 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:31.632096 (+ 69us) server_negotiation.cc:407] Connection header received
0812 16:50:31.634272 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:31.634315 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:31.635075 (+ 760us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:31.636134 (+ 1059us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:31.637781 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:31.637791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:31.637814 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:31.637836 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:31.637948 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:31.637985 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:31.638289 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:31.638537 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 16:50:31.638818 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":448,"threads_started":1}
I20260812 16:50:31.646133 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:31.648376 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0: Bootstrap starting.
I20260812 16:50:31.656056 508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:31.658162 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:31.654613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35843 (local address 127.0.0.1:34465)
0812 16:50:31.654864 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:31.654870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:31.655462 (+ 592us) server_negotiation.cc:407] Connection header received
0812 16:50:31.655707 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:31.655711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:31.655779 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:31.655901 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:31.657196 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:31.657201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:31.657204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:31.657227 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:31.657279 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:31.657297 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:31.657413 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:31.657606 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 16:50:31.657680 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 16:50:31.658161 508 log.cc:824] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:31.662037 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0: No bootstrap required, opened a new log
I20260812 16:50:31.672315 508 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4400f315ca14aecaa06a40cc87487b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34465 } }
I20260812 16:50:31.673008 508 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:31.673151 508 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4400f315ca14aecaa06a40cc87487b0, State: Initialized, Role: FOLLOWER
I20260812 16:50:31.674016 508 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4400f315ca14aecaa06a40cc87487b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34465 } }
I20260812 16:50:31.674280 508 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:31.674412 508 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:31.674608 508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:31.678171 508 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4400f315ca14aecaa06a40cc87487b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34465 } }
I20260812 16:50:31.679131 508 leader_election.cc:302] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4400f315ca14aecaa06a40cc87487b0; no voters:
I20260812 16:50:31.680070 508 leader_election.cc:288] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:31.680379 522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:31.681394 522 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [term 1 LEADER]: Becoming Leader. State: Replica: e4400f315ca14aecaa06a40cc87487b0, State: Running, Role: LEADER
I20260812 16:50:31.682148 522 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4400f315ca14aecaa06a40cc87487b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34465 } }
I20260812 16:50:31.682871 508 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:31.686791 528 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4400f315ca14aecaa06a40cc87487b0. Latest consensus state: current_term: 1 leader_uuid: "e4400f315ca14aecaa06a40cc87487b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4400f315ca14aecaa06a40cc87487b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34465 } } }
I20260812 16:50:31.686765 525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4400f315ca14aecaa06a40cc87487b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4400f315ca14aecaa06a40cc87487b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34465 } } }
I20260812 16:50:31.687422 525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:31.687422 528 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:31.689404 540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:31.697820 540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:31.709810 540 catalog_manager.cc:1347] Generated new cluster ID: cf0340d6d05440c18ac91bb7bd255309
I20260812 16:50:31.709962 540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:31.740482 540 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:31.741160 540 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:31.751398 540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4400f315ca14aecaa06a40cc87487b0: Generated new TSK 0
I20260812 16:50:31.752790 540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:31.758240 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 393
W20260812 16:50:31.759857 503 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53589
W20260812 16:50:31.761070 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 393: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:31.761279 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 393
2026-08-12T16:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:31 dist-test-slave-1vmg krb5kdc[561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:31 dist-test-slave-1vmg krb5kdc[561](info): set up 2 sockets
Aug 12 16:50:31 dist-test-slave-1vmg krb5kdc[561](info): commencing operation
krb5kdc: starting...
W20260812 16:50:33.114609 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:33 dist-test-slave-1vmg krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:33.513213 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:33.927495 586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:33.927814 586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:33.927969 586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:33.935781 586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:33.935945 586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:33.935994 586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:33.936049 586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:33.936085 586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:33.946956 586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:33.948601 586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:33.950073 586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:33.963203 594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:33.963392 593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:33.963191 599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:33.964227 586 server_base.cc:1034] running on GCE node
I20260812 16:50:33.965179 586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:33.966854 586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:33.968133 586 hybrid_clock.cc:648] HybridClock initialized: now 1786553433968077 us; error 61 us; skew 500 ppm
I20260812 16:50:33.968540 586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:33 dist-test-slave-1vmg krb5kdc[561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:33.972800 586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:33.975333 586 webserver.cc:460] Webserver started at http://127.0.0.1:46497/ using document root <none> and password file <none>
I20260812 16:50:33.976526 586 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:33.976802 586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:33.977454 586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:33.981415 586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "c6b317320aec40458c18837a45a112a6"
format_stamp: "Formatted at 2026-08-12 16:50:33 on dist-test-slave-1vmg"
server_key: "9201db39c01154abf0e2af4cd55cbd5c"
server_key_iv: "3006e6ff276030db1ef2085ed756aa55"
server_key_version: "encryptionkey@0"
I20260812 16:50:33.982354 586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "c6b317320aec40458c18837a45a112a6"
format_stamp: "Formatted at 2026-08-12 16:50:33 on dist-test-slave-1vmg"
server_key: "9201db39c01154abf0e2af4cd55cbd5c"
server_key_iv: "3006e6ff276030db1ef2085ed756aa55"
server_key_version: "encryptionkey@0"
I20260812 16:50:33.988184 586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:50:33.992210 611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:33.993944 586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 16:50:33.994180 586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "c6b317320aec40458c18837a45a112a6"
format_stamp: "Formatted at 2026-08-12 16:50:33 on dist-test-slave-1vmg"
server_key: "9201db39c01154abf0e2af4cd55cbd5c"
server_key_iv: "3006e6ff276030db1ef2085ed756aa55"
server_key_version: "encryptionkey@0"
I20260812 16:50:33.994520 586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:34.014385 586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:34.020021 586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:34.020367 586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:34.040880 586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43403
I20260812 16:50:34.040923 703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43403 every 8 connection(s)
I20260812 16:50:34.042402 586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:34.046557 709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:34.046684 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 586
I20260812 16:50:34.046880 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:34.047353 29644 external_mini_cluster.cc:1428] Setting key b82bf113ea3b7e81dac88566ff769776
I20260812 16:50:34.062676 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6: Bootstrap starting.
Aug 12 16:50:34 dist-test-slave-1vmg krb5kdc[561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:50:34.066555 709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:34.068116 709 log.cc:824] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:34.071851 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6: No bootstrap required, opened a new log
I20260812 16:50:34.076867 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:34.049142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45567 (local address 127.0.0.1:43403)
0812 16:50:34.050246 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:34.050267 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:34.050323 (+ 56us) server_negotiation.cc:407] Connection header received
0812 16:50:34.052942 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:34.052990 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:34.053702 (+ 712us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:34.054273 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:34.056547 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:34.058590 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:34.060116 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:34.061250 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:34.061453 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:34.066405 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:34.066420 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:34.066447 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:34.066505 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:34.070603 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:34.072253 (+ 1650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:34.072262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:34.072271 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:34.072535 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:34.073172 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:34.073177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:34.073180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:34.073993 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:34.074335 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:34.074795 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 16:50:34.075353 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":488,"threads_started":1}
I20260812 16:50:34.083097 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:34.083374 709 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b317320aec40458c18837a45a112a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43403 } }
I20260812 16:50:34.083890 709 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:34.083994 709 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b317320aec40458c18837a45a112a6, State: Initialized, Role: FOLLOWER
I20260812 16:50:34.084975 709 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b317320aec40458c18837a45a112a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43403 } }
I20260812 16:50:34.085340 709 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:34.085486 709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:34.085664 709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:34.089090 709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b317320aec40458c18837a45a112a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43403 } }
I20260812 16:50:34.089871 709 leader_election.cc:302] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b317320aec40458c18837a45a112a6; no voters:
I20260812 16:50:34.090626 709 leader_election.cc:288] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:34.090754 724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:34.092648 724 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [term 1 LEADER]: Becoming Leader. State: Replica: c6b317320aec40458c18837a45a112a6, State: Running, Role: LEADER
I20260812 16:50:34.093683 709 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:34.093600 724 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b317320aec40458c18837a45a112a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43403 } }
I20260812 16:50:34.097259 729 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b317320aec40458c18837a45a112a6. Latest consensus state: current_term: 1 leader_uuid: "c6b317320aec40458c18837a45a112a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b317320aec40458c18837a45a112a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43403 } } }
I20260812 16:50:34.098558 729 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:34.097987 725 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b317320aec40458c18837a45a112a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b317320aec40458c18837a45a112a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43403 } } }
I20260812 16:50:34.099687 725 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:34.101378 740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:34.110073 740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:34.116612 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:34.089094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60533 (local address 127.0.0.1:43403)
0812 16:50:34.089306 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:34.089312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:34.089705 (+ 393us) server_negotiation.cc:407] Connection header received
0812 16:50:34.089867 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:34.089872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:34.089963 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:34.090094 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:34.092406 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:34.094076 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:34.096296 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:34.097327 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:34.097520 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:34.105621 (+ 8101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:34.105630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:34.105634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:34.105692 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:34.110079 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:34.111623 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:34.111629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:34.111633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:34.111849 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:34.112578 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:34.112585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:34.112588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:34.113336 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:34.113955 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:34.115834 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0812 16:50:34.116182 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:50:34.123517 740 catalog_manager.cc:1347] Generated new cluster ID: 3c7d721cce674a3d8d2f621d1970a19e
I20260812 16:50:34.123663 740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:34.154054 740 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:34.156143 740 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:34.168073 740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b317320aec40458c18837a45a112a6: Generated new TSK 0
I20260812 16:50:34.169453 740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:34.181545 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 586
I20260812 16:50:34.182883 698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:34.288506 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 586
2026-08-12T16:50:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2531 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:34 dist-test-slave-1vmg krb5kdc[766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:34 dist-test-slave-1vmg krb5kdc[766](info): set up 2 sockets
Aug 12 16:50:34 dist-test-slave-1vmg krb5kdc[766](info): commencing operation
krb5kdc: starting...
W20260812 16:50:35.671226 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:35 dist-test-slave-1vmg krb5kdc[766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:36.081771 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:36.499306 796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:36.499670 796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:36.499807 796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:36.508227 796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:36.508384 796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:36.508450 796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:36.508543 796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:36.508687 796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:36.520112 796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:36.522496 796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:36.524624 796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:36.540087 806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:36.540410 807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:36.540936 810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:36.541163 796 server_base.cc:1034] running on GCE node
I20260812 16:50:36.542109 796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:36.543915 796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:36.545185 796 hybrid_clock.cc:648] HybridClock initialized: now 1786553436545104 us; error 93 us; skew 500 ppm
I20260812 16:50:36.545585 796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:36 dist-test-slave-1vmg krb5kdc[766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:36.549140 796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:36.550799 796 webserver.cc:460] Webserver started at http://127.0.0.1:41771/ using document root <none> and password file <none>
I20260812 16:50:36.551594 796 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:36.551728 796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:36.552035 796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:36.555437 796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "523d206b207e49d4aac81e1b6f7bef17"
format_stamp: "Formatted at 2026-08-12 16:50:36 on dist-test-slave-1vmg"
server_key: "1a9016fc16754debc11417d2a02d751e"
server_key_iv: "1d197d90700a9d4b2d0128a979a84227"
server_key_version: "encryptionkey@0"
I20260812 16:50:36.556365 796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "523d206b207e49d4aac81e1b6f7bef17"
format_stamp: "Formatted at 2026-08-12 16:50:36 on dist-test-slave-1vmg"
server_key: "1a9016fc16754debc11417d2a02d751e"
server_key_iv: "1d197d90700a9d4b2d0128a979a84227"
server_key_version: "encryptionkey@0"
I20260812 16:50:36.563591 796 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 16:50:36.568264 822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:36.570623 796 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:50:36.570897 796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "523d206b207e49d4aac81e1b6f7bef17"
format_stamp: "Formatted at 2026-08-12 16:50:36 on dist-test-slave-1vmg"
server_key: "1a9016fc16754debc11417d2a02d751e"
server_key_iv: "1d197d90700a9d4b2d0128a979a84227"
server_key_version: "encryptionkey@0"
I20260812 16:50:36.571174 796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:36.597913 796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:36.603992 796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:36.604342 796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:36.629823 907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42127 every 8 connection(s)
I20260812 16:50:36.630048 796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42127
I20260812 16:50:36.631978 796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:36.637048 908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:36.640162 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 796
I20260812 16:50:36.640348 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:36.640811 29644 external_mini_cluster.cc:1428] Setting key 30ba3cd63c5f67c1eb3e3df88a075f34
I20260812 16:50:36.652170 908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17: Bootstrap starting.
I20260812 16:50:36.656432 908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:36.657922 908 log.cc:824] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:36.658972 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:36.642894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51823 (local address 127.0.0.1:42127)
0812 16:50:36.643803 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:36.643820 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:36.643883 (+ 63us) server_negotiation.cc:407] Connection header received
0812 16:50:36.646322 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:36.646358 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:36.647191 (+ 833us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:36.648080 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:36.649001 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:36.651240 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:36.653149 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:36.654274 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:36.654514 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:36.655799 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:36.655811 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:36.655831 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:36.656609 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:36.656721 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:36.656759 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:36.656923 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:36.657187 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 16:50:36.657488 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":345,"threads_started":1}
I20260812 16:50:36.662293 908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17: No bootstrap required, opened a new log
I20260812 16:50:36.663169 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:36.672847 908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d206b207e49d4aac81e1b6f7bef17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260812 16:50:36.673475 908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:36.673597 908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523d206b207e49d4aac81e1b6f7bef17, State: Initialized, Role: FOLLOWER
I20260812 16:50:36.674309 908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d206b207e49d4aac81e1b6f7bef17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260812 16:50:36.674574 908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:36.674728 908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:36.674942 908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:36.678643 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:36.668994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41129 (local address 127.0.0.1:42127)
0812 16:50:36.669210 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:36.669216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:36.669823 (+ 607us) server_negotiation.cc:407] Connection header received
0812 16:50:36.669979 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:36.669984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:36.670090 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:36.670216 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:36.670864 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:36.672001 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:36.674171 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:36.674699 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:36.674827 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:36.676218 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:36.676226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:36.676229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:36.676446 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:36.676626 (+ 180us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:36.676665 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:36.676760 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:36.677744 (+ 984us) server_negotiation.cc:299] Negotiation successful
0812 16:50:36.678112 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:50:36.678870 908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d206b207e49d4aac81e1b6f7bef17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260812 16:50:36.679570 908 leader_election.cc:302] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523d206b207e49d4aac81e1b6f7bef17; no voters:
I20260812 16:50:36.680248 908 leader_election.cc:288] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:36.680388 926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:36.681070 926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [term 1 LEADER]: Becoming Leader. State: Replica: 523d206b207e49d4aac81e1b6f7bef17, State: Running, Role: LEADER
I20260812 16:50:36.681689 926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d206b207e49d4aac81e1b6f7bef17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260812 16:50:36.682533 908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:36.684564 930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523d206b207e49d4aac81e1b6f7bef17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d206b207e49d4aac81e1b6f7bef17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } } }
I20260812 16:50:36.685087 930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:36.684598 931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523d206b207e49d4aac81e1b6f7bef17. Latest consensus state: current_term: 1 leader_uuid: "523d206b207e49d4aac81e1b6f7bef17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d206b207e49d4aac81e1b6f7bef17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } } }
I20260812 16:50:36.685856 931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:36.687067 938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:36.693729 938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:36.706138 938 catalog_manager.cc:1347] Generated new cluster ID: 9bc1eba3d2724d58a7b676468b82f734
I20260812 16:50:36.706297 938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:36.716995 938 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:36.718410 938 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:36.726150 938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523d206b207e49d4aac81e1b6f7bef17: Generated new TSK 0
I20260812 16:50:36.727506 938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:36.739795 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 796
W20260812 16:50:36.741413 900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51823
W20260812 16:50:36.742517 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:36.742744 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 796
2026-08-12T16:50:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:36 dist-test-slave-1vmg krb5kdc[964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:36 dist-test-slave-1vmg krb5kdc[964](info): set up 2 sockets
Aug 12 16:50:36 dist-test-slave-1vmg krb5kdc[964](info): commencing operation
krb5kdc: starting...
W20260812 16:50:38.117163 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.000s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:38 dist-test-slave-1vmg krb5kdc[964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:38.536624 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:38.933653 990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:38.934005 990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:38.934135 990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:38.942462 990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:38.942677 990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:38.942763 990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:38.942855 990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:38.942903 990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:38.956187 990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:38.957808 990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:38.959347 990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:38.976231 1000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:38.976287 997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:38.976238 1003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:38.978055 990 server_base.cc:1034] running on GCE node
I20260812 16:50:38.979522 990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:38.981302 990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:38.982582 990 hybrid_clock.cc:648] HybridClock initialized: now 1786553438982519 us; error 51 us; skew 500 ppm
I20260812 16:50:38.983078 990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:38 dist-test-slave-1vmg krb5kdc[964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:38.987443 990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:38.990279 990 webserver.cc:460] Webserver started at http://127.0.0.1:44255/ using document root <none> and password file <none>
I20260812 16:50:38.991207 990 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:38.991358 990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:38.991696 990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:38.995262 990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "0287738156ad4e2db0711e5789746f9c"
format_stamp: "Formatted at 2026-08-12 16:50:38 on dist-test-slave-1vmg"
server_key: "6ae4cfef18caf662e06fa0b8a47b69c3"
server_key_iv: "01f26c36084d9dbbd8482e0c4ef316e4"
server_key_version: "encryptionkey@0"
I20260812 16:50:38.996189 990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "0287738156ad4e2db0711e5789746f9c"
format_stamp: "Formatted at 2026-08-12 16:50:38 on dist-test-slave-1vmg"
server_key: "6ae4cfef18caf662e06fa0b8a47b69c3"
server_key_iv: "01f26c36084d9dbbd8482e0c4ef316e4"
server_key_version: "encryptionkey@0"
I20260812 16:50:39.002647 990 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:50:39.007525 1015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:39.009569 990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:50:39.009826 990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "0287738156ad4e2db0711e5789746f9c"
format_stamp: "Formatted at 2026-08-12 16:50:38 on dist-test-slave-1vmg"
server_key: "6ae4cfef18caf662e06fa0b8a47b69c3"
server_key_iv: "01f26c36084d9dbbd8482e0c4ef316e4"
server_key_version: "encryptionkey@0"
I20260812 16:50:39.010063 990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:39.017019 990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:39.023955 990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:39.024669 990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:39.050735 990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42155
I20260812 16:50:39.050846 1105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42155 every 8 connection(s)
I20260812 16:50:39.052502 990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:39.059397 1106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:39.062619 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 990
I20260812 16:50:39.062834 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:39.063344 29644 external_mini_cluster.cc:1428] Setting key 40cee5c532e0dc48ca458a928e5143e9
I20260812 16:50:39.074157 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:39.065509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53987 (local address 127.0.0.1:42155)
0812 16:50:39.066504 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:39.066522 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:39.066582 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:50:39.069150 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:39.069200 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:39.069784 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:39.070521 (+ 737us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:39.071853 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:39.071862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:39.071885 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:39.071903 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:39.071983 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:39.072007 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:39.072270 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:39.072456 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 16:50:39.072713 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":433,"threads_started":1}
I20260812 16:50:39.077421 1106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c: Bootstrap starting.
I20260812 16:50:39.079485 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:39.081926 1106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:39.083372 1106 log.cc:824] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:39.088563 1106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c: No bootstrap required, opened a new log
I20260812 16:50:39.088687 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:39.084520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53519 (local address 127.0.0.1:42155)
0812 16:50:39.084718 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:39.084870 (+ 152us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:39.084906 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:50:39.085122 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:39.085128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:39.085207 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:39.085388 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:39.087508 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:39.087515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:39.087519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:39.087544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:39.087610 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:39.087630 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:39.087790 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:39.088052 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 16:50:39.088135 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 16:50:39.100143 1106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0287738156ad4e2db0711e5789746f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260812 16:50:39.100613 1106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:39.100715 1106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0287738156ad4e2db0711e5789746f9c, State: Initialized, Role: FOLLOWER
I20260812 16:50:39.101415 1106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0287738156ad4e2db0711e5789746f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260812 16:50:39.101679 1106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:39.101818 1106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:39.101977 1106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:39.104789 1106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0287738156ad4e2db0711e5789746f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260812 16:50:39.105393 1106 leader_election.cc:302] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0287738156ad4e2db0711e5789746f9c; no voters:
I20260812 16:50:39.106212 1106 leader_election.cc:288] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:39.106587 1118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:39.107300 1118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [term 1 LEADER]: Becoming Leader. State: Replica: 0287738156ad4e2db0711e5789746f9c, State: Running, Role: LEADER
I20260812 16:50:39.107915 1118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0287738156ad4e2db0711e5789746f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260812 16:50:39.108469 1106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:39.111085 1120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0287738156ad4e2db0711e5789746f9c. Latest consensus state: current_term: 1 leader_uuid: "0287738156ad4e2db0711e5789746f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0287738156ad4e2db0711e5789746f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } } }
I20260812 16:50:39.111853 1120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:39.111064 1119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0287738156ad4e2db0711e5789746f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0287738156ad4e2db0711e5789746f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } } }
I20260812 16:50:39.112522 1119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:39.112830 1131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:39.120757 1131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:39.134186 1131 catalog_manager.cc:1347] Generated new cluster ID: dfd624d43f2147008720a08c76850ea6
I20260812 16:50:39.134338 1131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:39.151954 1131 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:39.152686 1131 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:39.161872 1131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0287738156ad4e2db0711e5789746f9c: Generated new TSK 0
I20260812 16:50:39.162819 1131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:39.181432 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 990
W20260812 16:50:39.183475 1098 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53987
W20260812 16:50:39.184453 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:39.184639 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 990
2026-08-12T16:50:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:39 dist-test-slave-1vmg krb5kdc[1146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:39 dist-test-slave-1vmg krb5kdc[1146](info): set up 2 sockets
Aug 12 16:50:39 dist-test-slave-1vmg krb5kdc[1146](info): commencing operation
krb5kdc: starting...
W20260812 16:50:40.612841 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.370s user 0.002s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:40 dist-test-slave-1vmg krb5kdc[1146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:41.059429 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:41.547183 1163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:41.547508 1163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:41.547618 1163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:41.558033 1163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:41.558229 1163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:41.558295 1163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:41.558375 1163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:41.558429 1163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:41.572914 1163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:41.575224 1163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:41.577667 1163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:41.597958 1172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:41.597936 1176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:41.597936 1179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:41.600541 1163 server_base.cc:1034] running on GCE node
I20260812 16:50:41.601600 1163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:41.603482 1163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:41.604790 1163 hybrid_clock.cc:648] HybridClock initialized: now 1786553441604736 us; error 52 us; skew 500 ppm
I20260812 16:50:41.605273 1163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:41 dist-test-slave-1vmg krb5kdc[1146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:41.609355 1163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:41.611590 1163 webserver.cc:460] Webserver started at http://127.0.0.1:35175/ using document root <none> and password file <none>
I20260812 16:50:41.612445 1163 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:41.612681 1163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:41.613397 1163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:41.619410 1163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "c1339818f7c440c4b296424d0b496154"
format_stamp: "Formatted at 2026-08-12 16:50:41 on dist-test-slave-1vmg"
server_key: "e81eaf47ce65ddf660b6bbc35149399d"
server_key_iv: "f62021ee377c75be06502659b1e6cf62"
server_key_version: "encryptionkey@0"
I20260812 16:50:41.620925 1163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "c1339818f7c440c4b296424d0b496154"
format_stamp: "Formatted at 2026-08-12 16:50:41 on dist-test-slave-1vmg"
server_key: "e81eaf47ce65ddf660b6bbc35149399d"
server_key_iv: "f62021ee377c75be06502659b1e6cf62"
server_key_version: "encryptionkey@0"
I20260812 16:50:41.632731 1163 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260812 16:50:41.640218 1190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:41.642634 1163 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260812 16:50:41.642897 1163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "c1339818f7c440c4b296424d0b496154"
format_stamp: "Formatted at 2026-08-12 16:50:41 on dist-test-slave-1vmg"
server_key: "e81eaf47ce65ddf660b6bbc35149399d"
server_key_iv: "f62021ee377c75be06502659b1e6cf62"
server_key_version: "encryptionkey@0"
I20260812 16:50:41.643262 1163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:41.670075 1163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:41.680757 1163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:41.681160 1163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:41.706674 1163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42889
I20260812 16:50:41.706779 1281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42889 every 8 connection(s)
I20260812 16:50:41.708313 1163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:41.713562 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1163
I20260812 16:50:41.714030 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:41.714692 29644 external_mini_cluster.cc:1428] Setting key c234856de44ff7dc4a9c91e97b6313b7
I20260812 16:50:41.716122 1282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 16:50:41 dist-test-slave-1vmg krb5kdc[1146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:50:41.734589 1282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154: Bootstrap starting.
I20260812 16:50:41.740365 1282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:41.742348 1282 log.cc:824] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:41.746142 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:41.718223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:54305 (local address 127.0.0.1:42889)
0812 16:50:41.719679 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:41.719698 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:41.719782 (+ 84us) server_negotiation.cc:407] Connection header received
0812 16:50:41.721620 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:41.721657 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:41.722259 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:41.722761 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:41.724487 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:41.725933 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:41.727298 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:41.728049 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:41.728249 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:41.734395 (+ 6146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:41.734407 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:41.734428 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:41.734484 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:41.739659 (+ 5175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:41.740917 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:41.740933 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:41.740948 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:41.741405 (+ 457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:41.742168 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:41.742174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:41.742177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:41.743192 (+ 1015us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:41.743527 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:41.744293 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 16:50:41.744696 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":662,"threads_started":1}
I20260812 16:50:41.746524 1282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154: No bootstrap required, opened a new log
I20260812 16:50:41.752569 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:41.759217 1282 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1339818f7c440c4b296424d0b496154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260812 16:50:41.760041 1282 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:41.760179 1282 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1339818f7c440c4b296424d0b496154, State: Initialized, Role: FOLLOWER
I20260812 16:50:41.761219 1282 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1339818f7c440c4b296424d0b496154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260812 16:50:41.761538 1282 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:41.761690 1282 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:41.761863 1282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:41.765098 1282 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1339818f7c440c4b296424d0b496154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260812 16:50:41.765735 1282 leader_election.cc:302] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1339818f7c440c4b296424d0b496154; no voters:
I20260812 16:50:41.766460 1282 leader_election.cc:288] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:41.766578 1300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:41.767848 1300 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [term 1 LEADER]: Becoming Leader. State: Replica: c1339818f7c440c4b296424d0b496154, State: Running, Role: LEADER
I20260812 16:50:41.768579 1300 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1339818f7c440c4b296424d0b496154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260812 16:50:41.769562 1282 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:41.773991 1301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1339818f7c440c4b296424d0b496154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1339818f7c440c4b296424d0b496154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } } }
I20260812 16:50:41.774588 1301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:41.775821 1304 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1339818f7c440c4b296424d0b496154. Latest consensus state: current_term: 1 leader_uuid: "c1339818f7c440c4b296424d0b496154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1339818f7c440c4b296424d0b496154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } } }
I20260812 16:50:41.776309 1304 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:41.781558 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:41.759371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:34595 (local address 127.0.0.1:42889)
0812 16:50:41.759565 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:41.759571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:41.760239 (+ 668us) server_negotiation.cc:407] Connection header received
0812 16:50:41.760500 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:41.760507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:41.760610 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:41.760745 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:41.762774 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:41.764064 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:41.765478 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:41.766007 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:41.766132 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:41.769384 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:41.769407 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:41.769413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:41.769484 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:41.775164 (+ 5680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:41.776949 (+ 1785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:41.776955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:41.776959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:41.777199 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:41.777967 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:41.777972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:41.777975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:41.778578 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:41.780523 (+ 1945us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:41.780799 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 16:50:41.781056 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:50:41.784637 1311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:41.792153 1311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:41.805341 1311 catalog_manager.cc:1347] Generated new cluster ID: 8f0c6d670bce431eba362adbeeef83a9
I20260812 16:50:41.805476 1311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:41.819062 1311 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:41.821185 1311 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:41.833153 1311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1339818f7c440c4b296424d0b496154: Generated new TSK 0
I20260812 16:50:41.834412 1311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:41.846669 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1163
I20260812 16:50:41.848059 1274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:41.948216 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1163
W20260812 16:50:41.979543 1144 connection.cc:535] client connection to 127.0.0.1:42889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42889): Connection reset by peer (error 104)
2026-08-12T16:50:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2757 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:42 dist-test-slave-1vmg krb5kdc[1347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:42 dist-test-slave-1vmg krb5kdc[1347](info): set up 2 sockets
Aug 12 16:50:42 dist-test-slave-1vmg krb5kdc[1347](info): commencing operation
krb5kdc: starting...
W20260812 16:50:43.351123 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.350s user 0.006s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:43 dist-test-slave-1vmg krb5kdc[1347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:43.763163 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:44.170744 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:44.171298 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:44.171469 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:44.179512 1368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:44.179700 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:44.179804 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:44.179893 1368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:44.179955 1368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:44.192185 1368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:44.194211 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:44.195885 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:44.209607 1378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:44.209810 1380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:44.209674 1384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:44.210757 1368 server_base.cc:1034] running on GCE node
I20260812 16:50:44.211789 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:44.213599 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:44.217151 1368 hybrid_clock.cc:648] HybridClock initialized: now 1786553444217021 us; error 93 us; skew 500 ppm
I20260812 16:50:44.217684 1368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:44 dist-test-slave-1vmg krb5kdc[1347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:44.228976 1368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:44.231206 1368 webserver.cc:460] Webserver started at http://127.0.0.1:43675/ using document root <none> and password file <none>
I20260812 16:50:44.232010 1368 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:44.232161 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:44.232503 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:44.235999 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "b56e86be00304138b50ca65f0f38e5ce"
format_stamp: "Formatted at 2026-08-12 16:50:44 on dist-test-slave-1vmg"
server_key: "6d20ed7db625e82a1cd09030661b8c18"
server_key_iv: "801c39b503fe0e121cb0507817c2caf8"
server_key_version: "encryptionkey@0"
I20260812 16:50:44.237064 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "b56e86be00304138b50ca65f0f38e5ce"
format_stamp: "Formatted at 2026-08-12 16:50:44 on dist-test-slave-1vmg"
server_key: "6d20ed7db625e82a1cd09030661b8c18"
server_key_iv: "801c39b503fe0e121cb0507817c2caf8"
server_key_version: "encryptionkey@0"
I20260812 16:50:44.243541 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 16:50:44.247813 1393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:44.249630 1368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:50:44.249874 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "b56e86be00304138b50ca65f0f38e5ce"
format_stamp: "Formatted at 2026-08-12 16:50:44 on dist-test-slave-1vmg"
server_key: "6d20ed7db625e82a1cd09030661b8c18"
server_key_iv: "801c39b503fe0e121cb0507817c2caf8"
server_key_version: "encryptionkey@0"
I20260812 16:50:44.250319 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:44.271678 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:44.277457 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:44.277794 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:44.299934 1368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35997
I20260812 16:50:44.299954 1484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35997 every 8 connection(s)
I20260812 16:50:44.301579 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:44.306083 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1368
I20260812 16:50:44.306223 1487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:44.306299 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:44.306898 29644 external_mini_cluster.cc:1428] Setting key 470ac7579c0fc20036faba1a4c31a632
I20260812 16:50:44.322225 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:44.309068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:48407 (local address 127.0.0.1:35997)
0812 16:50:44.309980 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:44.309993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:44.310041 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:50:44.312255 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:44.312288 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:44.312796 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:44.313229 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:44.313880 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:44.315421 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:44.316917 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:44.317735 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:44.317950 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:44.319396 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:44.319405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:44.319423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:44.320201 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:44.320312 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:44.320344 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:44.320547 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:44.320901 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 16:50:44.321106 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":387,"threads_started":1}
I20260812 16:50:44.323979 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce: Bootstrap starting.
I20260812 16:50:44.327728 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:44.327899 1487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:44.329556 1487 log.cc:824] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:44.335793 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce: No bootstrap required, opened a new log
I20260812 16:50:44.341235 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:44.332917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:53987 (local address 127.0.0.1:35997)
0812 16:50:44.333160 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:44.333168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:44.333414 (+ 246us) server_negotiation.cc:407] Connection header received
0812 16:50:44.333635 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:44.333642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:44.333753 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:44.333930 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:44.334689 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:44.335846 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:44.337582 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:44.338691 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:44.338854 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:44.340155 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:44.340162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:44.340167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:44.340199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:44.340255 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:44.340277 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:44.340417 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:44.340822 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 16:50:44.340896 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 16:50:44.345847 1487 raft_consensus.cc:348] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e86be00304138b50ca65f0f38e5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 16:50:44.346323 1487 raft_consensus.cc:374] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:44.346439 1487 raft_consensus.cc:729] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56e86be00304138b50ca65f0f38e5ce, State: Initialized, Role: FOLLOWER
I20260812 16:50:44.347139 1487 consensus_queue.cc:260] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e86be00304138b50ca65f0f38e5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 16:50:44.347397 1487 raft_consensus.cc:388] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:44.347517 1487 raft_consensus.cc:482] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:44.347690 1487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:44.350414 1487 raft_consensus.cc:504] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e86be00304138b50ca65f0f38e5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 16:50:44.351485 1487 leader_election.cc:302] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b56e86be00304138b50ca65f0f38e5ce; no voters:
I20260812 16:50:44.352435 1487 leader_election.cc:288] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:44.352743 1512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:44.354094 1512 raft_consensus.cc:686] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [term 1 LEADER]: Becoming Leader. State: Replica: b56e86be00304138b50ca65f0f38e5ce, State: Running, Role: LEADER
I20260812 16:50:44.354804 1512 consensus_queue.cc:237] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e86be00304138b50ca65f0f38e5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 16:50:44.355690 1487 sys_catalog.cc:564] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:44.359202 1513 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b56e86be00304138b50ca65f0f38e5ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e86be00304138b50ca65f0f38e5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } } }
I20260812 16:50:44.359774 1513 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:44.360267 1514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b56e86be00304138b50ca65f0f38e5ce. Latest consensus state: current_term: 1 leader_uuid: "b56e86be00304138b50ca65f0f38e5ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e86be00304138b50ca65f0f38e5ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } } }
I20260812 16:50:44.360657 1514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:44.361721 1527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:44.368680 1527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:44.379997 1527 catalog_manager.cc:1347] Generated new cluster ID: ca97c448f2144dbe813435bfc8f03b5a
I20260812 16:50:44.380154 1527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:44.400123 1527 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:44.402300 1527 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:44.416525 1527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b56e86be00304138b50ca65f0f38e5ce: Generated new TSK 0
I20260812 16:50:44.418161 1527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:44.435802 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1368
W20260812 16:50:44.437177 1475 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:48407
W20260812 16:50:44.438032 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:44.438164 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1368
W20260812 16:50:44.468015 1343 connection.cc:535] client connection to 127.0.0.1:35997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35997): Connection reset by peer (error 104)
2026-08-12T16:50:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:44 dist-test-slave-1vmg krb5kdc[1545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:44 dist-test-slave-1vmg krb5kdc[1545](info): set up 2 sockets
Aug 12 16:50:44 dist-test-slave-1vmg krb5kdc[1545](info): commencing operation
krb5kdc: starting...
W20260812 16:50:45.816520 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.003s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:46 dist-test-slave-1vmg krb5kdc[1545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:46.235169 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:46.666380 1563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:46.666725 1563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:46.666833 1563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:46.677881 1563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:46.678078 1563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:46.678141 1563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:46.678220 1563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:46.678359 1563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:46.690462 1563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:46.692914 1563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:46.695537 1563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:46.710335 1571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:46.710350 1575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:46.710381 1572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:46.711272 1563 server_base.cc:1034] running on GCE node
I20260812 16:50:46.712224 1563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:46.714766 1563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:46.716327 1563 hybrid_clock.cc:648] HybridClock initialized: now 1786553446716274 us; error 66 us; skew 500 ppm
I20260812 16:50:46.716874 1563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:46 dist-test-slave-1vmg krb5kdc[1545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:46.722198 1563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:46.724028 1563 webserver.cc:460] Webserver started at http://127.0.0.1:40971/ using document root <none> and password file <none>
I20260812 16:50:46.724807 1563 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:46.724931 1563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:46.725220 1563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:46.728724 1563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "892634ebc8994cdab9f83ff8fa837ecd"
format_stamp: "Formatted at 2026-08-12 16:50:46 on dist-test-slave-1vmg"
server_key: "3624e26e7a4027c77a1d27615cf08f46"
server_key_iv: "fed77b8051abe15107149fdfcc3860e0"
server_key_version: "encryptionkey@0"
I20260812 16:50:46.729617 1563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "892634ebc8994cdab9f83ff8fa837ecd"
format_stamp: "Formatted at 2026-08-12 16:50:46 on dist-test-slave-1vmg"
server_key: "3624e26e7a4027c77a1d27615cf08f46"
server_key_iv: "fed77b8051abe15107149fdfcc3860e0"
server_key_version: "encryptionkey@0"
I20260812 16:50:46.738178 1563 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260812 16:50:46.743165 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:46.744932 1563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:50:46.745151 1563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "892634ebc8994cdab9f83ff8fa837ecd"
format_stamp: "Formatted at 2026-08-12 16:50:46 on dist-test-slave-1vmg"
server_key: "3624e26e7a4027c77a1d27615cf08f46"
server_key_iv: "fed77b8051abe15107149fdfcc3860e0"
server_key_version: "encryptionkey@0"
I20260812 16:50:46.745373 1563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:46.752295 1563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:46.759209 1563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:46.759595 1563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:46.783294 1563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44061
I20260812 16:50:46.783315 1674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44061 every 8 connection(s)
I20260812 16:50:46.784981 1563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:46.790064 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1563
I20260812 16:50:46.790156 1677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:46.790390 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:46.791438 29644 external_mini_cluster.cc:1428] Setting key 1c0ec844506a0ded50370d4b76daa56c
I20260812 16:50:46.804920 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:46.794802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:34897 (local address 127.0.0.1:44061)
0812 16:50:46.798047 (+ 3245us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:46.798066 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:46.798126 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:50:46.800052 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:46.800088 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:46.800648 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:46.801191 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:46.802634 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:46.802643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:46.802661 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:46.802683 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:46.802788 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:46.802819 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:46.803165 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:46.803340 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 16:50:46.803603 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2905,"thread_start_us":2442,"threads_started":1}
I20260812 16:50:46.810775 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:46.810849 1677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd: Bootstrap starting.
I20260812 16:50:46.815685 1677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:46.817610 1677 log.cc:824] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:46.820274 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:46.816497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:35885 (local address 127.0.0.1:44061)
0812 16:50:46.816887 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:46.816894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:46.817318 (+ 424us) server_negotiation.cc:407] Connection header received
0812 16:50:46.817637 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:46.817643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:46.817714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:46.817839 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:46.819307 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:46.819314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:46.819318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:46.819348 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:46.819422 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:46.819443 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:46.819602 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:46.819850 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 16:50:46.819935 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260812 16:50:46.822244 1677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd: No bootstrap required, opened a new log
I20260812 16:50:46.830564 1677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892634ebc8994cdab9f83ff8fa837ecd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260812 16:50:46.831223 1677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:46.831367 1677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892634ebc8994cdab9f83ff8fa837ecd, State: Initialized, Role: FOLLOWER
I20260812 16:50:46.833184 1677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892634ebc8994cdab9f83ff8fa837ecd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260812 16:50:46.833729 1677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:46.833918 1677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:46.834132 1677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:46.837152 1677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892634ebc8994cdab9f83ff8fa837ecd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260812 16:50:46.837970 1677 leader_election.cc:302] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892634ebc8994cdab9f83ff8fa837ecd; no voters:
I20260812 16:50:46.838822 1677 leader_election.cc:288] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:46.839054 1694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:46.840072 1694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [term 1 LEADER]: Becoming Leader. State: Replica: 892634ebc8994cdab9f83ff8fa837ecd, State: Running, Role: LEADER
I20260812 16:50:46.840839 1694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892634ebc8994cdab9f83ff8fa837ecd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260812 16:50:46.841578 1677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:46.844523 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892634ebc8994cdab9f83ff8fa837ecd. Latest consensus state: current_term: 1 leader_uuid: "892634ebc8994cdab9f83ff8fa837ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892634ebc8994cdab9f83ff8fa837ecd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } } }
I20260812 16:50:46.844507 1696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892634ebc8994cdab9f83ff8fa837ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892634ebc8994cdab9f83ff8fa837ecd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } } }
I20260812 16:50:46.845060 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:46.845129 1696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:46.848446 1712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:46.863031 1712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:46.875370 1712 catalog_manager.cc:1347] Generated new cluster ID: 7dcac5a4ec974d4a95b180dc60cce12b
I20260812 16:50:46.875633 1712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:46.911095 1712 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:46.911775 1712 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:46.922129 1712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892634ebc8994cdab9f83ff8fa837ecd: Generated new TSK 0
I20260812 16:50:46.923316 1712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:46.941282 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1563
W20260812 16:50:46.943115 1670 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:34897
W20260812 16:50:46.943902 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:46.944090 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1563
2026-08-12T16:50:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:47 dist-test-slave-1vmg krb5kdc[1739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:47 dist-test-slave-1vmg krb5kdc[1739](info): set up 2 sockets
Aug 12 16:50:47 dist-test-slave-1vmg krb5kdc[1739](info): commencing operation
krb5kdc: starting...
W20260812 16:50:48.356364 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.353s user 0.007s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:48 dist-test-slave-1vmg krb5kdc[1739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:48.759807 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:49.197897 1760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:49.198249 1760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:49.198347 1760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:49.207270 1760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:49.207473 1760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:49.207531 1760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:49.207597 1760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:49.207649 1760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:49.220664 1760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:49.222534 1760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:49.224071 1760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:49.243561 1771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:49.243520 1774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:49.244258 1767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:49.246012 1760 server_base.cc:1034] running on GCE node
I20260812 16:50:49.247016 1760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:49.248803 1760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:49.250285 1760 hybrid_clock.cc:648] HybridClock initialized: now 1786553449250221 us; error 64 us; skew 500 ppm
I20260812 16:50:49.251433 1760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:49 dist-test-slave-1vmg krb5kdc[1739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:49.260663 1760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:49.262724 1760 webserver.cc:460] Webserver started at http://127.0.0.1:34301/ using document root <none> and password file <none>
I20260812 16:50:49.263592 1760 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:49.263737 1760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:49.264037 1760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:49.267666 1760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "ff29e4a0270b413f86e6dbb274ef2b72"
format_stamp: "Formatted at 2026-08-12 16:50:49 on dist-test-slave-1vmg"
server_key: "b0fb9505b939222cb87e5b320b7a026e"
server_key_iv: "81bc625dc226b0458815af488ed0a8fa"
server_key_version: "encryptionkey@0"
I20260812 16:50:49.268869 1760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "ff29e4a0270b413f86e6dbb274ef2b72"
format_stamp: "Formatted at 2026-08-12 16:50:49 on dist-test-slave-1vmg"
server_key: "b0fb9505b939222cb87e5b320b7a026e"
server_key_iv: "81bc625dc226b0458815af488ed0a8fa"
server_key_version: "encryptionkey@0"
I20260812 16:50:49.275290 1760 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:50:49.280987 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:49.283106 1760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:50:49.283339 1760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "ff29e4a0270b413f86e6dbb274ef2b72"
format_stamp: "Formatted at 2026-08-12 16:50:49 on dist-test-slave-1vmg"
server_key: "b0fb9505b939222cb87e5b320b7a026e"
server_key_iv: "81bc625dc226b0458815af488ed0a8fa"
server_key_version: "encryptionkey@0"
I20260812 16:50:49.283581 1760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:49.304162 1760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:49.309563 1760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:49.310117 1760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:49.335273 1760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41577
I20260812 16:50:49.335332 1880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41577 every 8 connection(s)
I20260812 16:50:49.337466 1760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:49.343442 1882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:49.344152 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1760
I20260812 16:50:49.344388 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:49.345039 29644 external_mini_cluster.cc:1428] Setting key 9ad1bf2f931308069254711821502844
I20260812 16:50:49.360468 1882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72: Bootstrap starting.
Aug 12 16:50:49 dist-test-slave-1vmg krb5kdc[1739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:50:49.365415 1882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:49.366813 1882 log.cc:824] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:49.371215 1882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72: No bootstrap required, opened a new log
I20260812 16:50:49.376612 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:49.347795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:32945 (local address 127.0.0.1:41577)
0812 16:50:49.348907 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:49.348930 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:49.349002 (+ 72us) server_negotiation.cc:407] Connection header received
0812 16:50:49.351290 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:49.351331 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:49.352053 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:49.352553 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:49.354849 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:49.356434 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:49.358448 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:49.359347 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:49.359515 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:49.365291 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:49.365306 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:49.365327 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:49.365390 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:49.369633 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:49.371242 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:49.371253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:49.371263 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:49.371601 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:49.372404 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:49.372411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:49.372414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:49.373144 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:49.373482 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:49.374330 (+ 848us) server_negotiation.cc:299] Negotiation successful
0812 16:50:49.375026 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":476,"threads_started":1}
I20260812 16:50:49.382102 1882 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41577 } }
I20260812 16:50:49.382644 1882 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:49.382776 1882 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff29e4a0270b413f86e6dbb274ef2b72, State: Initialized, Role: FOLLOWER
I20260812 16:50:49.383496 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:49.383617 1882 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41577 } }
I20260812 16:50:49.383935 1882 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:49.384092 1882 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:49.384274 1882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:49.387431 1882 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41577 } }
I20260812 16:50:49.388203 1882 leader_election.cc:302] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff29e4a0270b413f86e6dbb274ef2b72; no voters:
I20260812 16:50:49.391321 1882 leader_election.cc:288] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:49.391484 1897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:49.394102 1882 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:49.395319 1897 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [term 1 LEADER]: Becoming Leader. State: Replica: ff29e4a0270b413f86e6dbb274ef2b72, State: Running, Role: LEADER
I20260812 16:50:49.396479 1897 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41577 } }
I20260812 16:50:49.400864 1898 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41577 } } }
I20260812 16:50:49.401197 1899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff29e4a0270b413f86e6dbb274ef2b72. Latest consensus state: current_term: 1 leader_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff29e4a0270b413f86e6dbb274ef2b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41577 } } }
I20260812 16:50:49.401511 1898 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:49.401603 1899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:49.411123 1906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:49.414587 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:49.388719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:36891 (local address 127.0.0.1:41577)
0812 16:50:49.388949 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:49.388960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:49.389562 (+ 602us) server_negotiation.cc:407] Connection header received
0812 16:50:49.389766 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:49.389773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:49.389891 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:49.390082 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:49.393303 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:49.394492 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:49.397199 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:49.398039 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:49.398238 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:49.404771 (+ 6533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:49.404781 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:49.404787 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:49.404847 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:49.409168 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:49.411157 (+ 1989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:49.411165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:49.411168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:49.411549 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:49.412332 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:49.412337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:49.412340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:49.412939 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:49.413322 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:49.413883 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 16:50:49.414136 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:50:49.421506 1906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:49.432983 1906 catalog_manager.cc:1347] Generated new cluster ID: 7ca29b7d8f724025b9e507861c5a325c
I20260812 16:50:49.433135 1906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:49.454526 1906 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:49.457464 1906 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:49.471560 1906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff29e4a0270b413f86e6dbb274ef2b72: Generated new TSK 0
I20260812 16:50:49.472546 1906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:49.485467 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1760
I20260812 16:50:49.486801 1873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:49.589833 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1760
W20260812 16:50:49.623399 1734 connection.cc:535] client connection to 127.0.0.1:41577 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41577): Connection reset by peer (error 104)
2026-08-12T16:50:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2643 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:49 dist-test-slave-1vmg krb5kdc[1932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:49 dist-test-slave-1vmg krb5kdc[1932](info): set up 2 sockets
Aug 12 16:50:49 dist-test-slave-1vmg krb5kdc[1932](info): commencing operation
krb5kdc: starting...
W20260812 16:50:50.975850 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.002s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:51 dist-test-slave-1vmg krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:51.383282 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:51.848838 1961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:51.849537 1961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:51.849735 1961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:51.860625 1961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:51.860922 1961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:51.861047 1961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:51.861135 1961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:51.861209 1961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:51.879294 1961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:51.881677 1961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:51.883903 1961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:51.900699 1970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:51.901223 1975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:51.901288 1968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:51.901962 1961 server_base.cc:1034] running on GCE node
I20260812 16:50:51.902946 1961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:51.904663 1961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:51.905951 1961 hybrid_clock.cc:648] HybridClock initialized: now 1786553451905899 us; error 39 us; skew 500 ppm
I20260812 16:50:51.906370 1961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:51 dist-test-slave-1vmg krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:51.910832 1961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:51.913817 1961 webserver.cc:460] Webserver started at http://127.0.0.1:40897/ using document root <none> and password file <none>
I20260812 16:50:51.914970 1961 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:51.915163 1961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:51.915587 1961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:51.920652 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "50ffcb09693144d6b4829928778a48a0"
format_stamp: "Formatted at 2026-08-12 16:50:51 on dist-test-slave-1vmg"
server_key: "a9881b6fade7418d7ab65d351ff9296c"
server_key_iv: "fef2288371d8eeef825e609ece8e9016"
server_key_version: "encryptionkey@0"
I20260812 16:50:51.921829 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "50ffcb09693144d6b4829928778a48a0"
format_stamp: "Formatted at 2026-08-12 16:50:51 on dist-test-slave-1vmg"
server_key: "a9881b6fade7418d7ab65d351ff9296c"
server_key_iv: "fef2288371d8eeef825e609ece8e9016"
server_key_version: "encryptionkey@0"
I20260812 16:50:51.929049 1961 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 16:50:51.935798 1993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:51.938421 1961 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260812 16:50:51.938710 1961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "50ffcb09693144d6b4829928778a48a0"
format_stamp: "Formatted at 2026-08-12 16:50:51 on dist-test-slave-1vmg"
server_key: "a9881b6fade7418d7ab65d351ff9296c"
server_key_iv: "fef2288371d8eeef825e609ece8e9016"
server_key_version: "encryptionkey@0"
I20260812 16:50:51.939073 1961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:51.956704 1961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:51.963510 1961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:51.963882 1961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:51.988336 1961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41069
I20260812 16:50:51.988493 2069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41069 every 8 connection(s)
I20260812 16:50:51.990103 1961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:51.993345 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 1961
I20260812 16:50:51.993602 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:51.994299 29644 external_mini_cluster.cc:1428] Setting key 83a2314587cd6ba7509c771f35d30346
I20260812 16:50:51.997186 2071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:52.013120 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:51.997431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:34635 (local address 127.0.0.1:41069)
0812 16:50:51.998703 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:51.998717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:51.998766 (+ 49us) server_negotiation.cc:407] Connection header received
0812 16:50:52.000827 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:52.000854 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:52.001367 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:52.001777 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:52.002660 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:52.004202 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:52.006339 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:52.007176 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:52.007328 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:52.008904 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:52.008914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:52.008935 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:52.009875 (+ 940us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:52.009993 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:52.010024 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:52.010182 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:52.011193 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0812 16:50:52.011622 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":770,"threads_started":1}
I20260812 16:50:52.014401 2071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0: Bootstrap starting.
I20260812 16:50:52.020999 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:52.021239 2071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:52.022424 2071 log.cc:824] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:52.026185 2071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0: No bootstrap required, opened a new log
I20260812 16:50:52.037063 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:52.025966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:33443 (local address 127.0.0.1:41069)
0812 16:50:52.026174 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:52.026182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:52.026568 (+ 386us) server_negotiation.cc:407] Connection header received
0812 16:50:52.026749 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:52.026755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:52.026857 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:52.027010 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:52.027611 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:52.028788 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:52.032462 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:52.033434 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:52.033630 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:52.035581 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:52.035590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:52.035595 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:52.035624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:52.035691 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:52.035720 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:52.035882 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:52.036410 (+ 528us) server_negotiation.cc:299] Negotiation successful
0812 16:50:52.036517 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:50:52.042420 2071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ffcb09693144d6b4829928778a48a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 16:50:52.043054 2071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:52.043183 2071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ffcb09693144d6b4829928778a48a0, State: Initialized, Role: FOLLOWER
I20260812 16:50:52.043884 2071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ffcb09693144d6b4829928778a48a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 16:50:52.044133 2071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:52.044288 2071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:52.044456 2071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:52.046991 2071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ffcb09693144d6b4829928778a48a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 16:50:52.047750 2071 leader_election.cc:302] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ffcb09693144d6b4829928778a48a0; no voters:
I20260812 16:50:52.048753 2071 leader_election.cc:288] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:52.049022 2083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:52.051666 2083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [term 1 LEADER]: Becoming Leader. State: Replica: 50ffcb09693144d6b4829928778a48a0, State: Running, Role: LEADER
I20260812 16:50:52.054476 2071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:52.055155 2083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ffcb09693144d6b4829928778a48a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 16:50:52.061468 2084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ffcb09693144d6b4829928778a48a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ffcb09693144d6b4829928778a48a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } } }
I20260812 16:50:52.061452 2085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ffcb09693144d6b4829928778a48a0. Latest consensus state: current_term: 1 leader_uuid: "50ffcb09693144d6b4829928778a48a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ffcb09693144d6b4829928778a48a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } } }
I20260812 16:50:52.062477 2084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:52.062477 2085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:52.065008 2097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:52.078903 2097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:52.100926 2097 catalog_manager.cc:1347] Generated new cluster ID: d1f00d62f06343879b919335bfdd5f54
I20260812 16:50:52.101079 2097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:52.131098 2097 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:52.133476 2097 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:52.141455 2097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ffcb09693144d6b4829928778a48a0: Generated new TSK 0
I20260812 16:50:52.142450 2097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:52.157873 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1961
W20260812 16:50:52.159883 2064 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:34635
W20260812 16:50:52.160634 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 1961: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:52.161023 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 1961
W20260812 16:50:52.191413 1929 connection.cc:535] client connection to 127.0.0.1:41069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41069): Connection reset by peer (error 104)
2026-08-12T16:50:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2564 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:52 dist-test-slave-1vmg krb5kdc[2119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:52 dist-test-slave-1vmg krb5kdc[2119](info): set up 2 sockets
Aug 12 16:50:52 dist-test-slave-1vmg krb5kdc[2119](info): commencing operation
krb5kdc: starting...
W20260812 16:50:53.517766 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.007s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:53 dist-test-slave-1vmg krb5kdc[2119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:50:53.920166 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:54.346189 2143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:54.346545 2143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:54.346674 2143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:54.355611 2143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:54.355770 2143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:54.355822 2143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:54.355897 2143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:54.355991 2143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:54.369089 2143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:54.370947 2143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:54.372548 2143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:54.386413 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:54.386375 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:54.386308 2154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:54.387089 2143 server_base.cc:1034] running on GCE node
I20260812 16:50:54.388216 2143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:54.390095 2143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:54.393523 2143 hybrid_clock.cc:648] HybridClock initialized: now 1786553454393428 us; error 149 us; skew 500 ppm
I20260812 16:50:54.394023 2143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:54 dist-test-slave-1vmg krb5kdc[2119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:54.402428 2143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:50:54.404330 2143 webserver.cc:460] Webserver started at http://127.0.0.1:45669/ using document root <none> and password file <none>
I20260812 16:50:54.405124 2143 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:54.405272 2143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:54.405614 2143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:54.409099 2143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "124259bc843743b79b575094d916131d"
format_stamp: "Formatted at 2026-08-12 16:50:54 on dist-test-slave-1vmg"
server_key: "0317a10dde6fd27192e66fa471cf5ee8"
server_key_iv: "60114306f95c6101227f1eb650e3ac9d"
server_key_version: "encryptionkey@0"
I20260812 16:50:54.409999 2143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "124259bc843743b79b575094d916131d"
format_stamp: "Formatted at 2026-08-12 16:50:54 on dist-test-slave-1vmg"
server_key: "0317a10dde6fd27192e66fa471cf5ee8"
server_key_iv: "60114306f95c6101227f1eb650e3ac9d"
server_key_version: "encryptionkey@0"
I20260812 16:50:54.416263 2143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 16:50:54.420591 2169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:54.423005 2143 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:50:54.423368 2143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "124259bc843743b79b575094d916131d"
format_stamp: "Formatted at 2026-08-12 16:50:54 on dist-test-slave-1vmg"
server_key: "0317a10dde6fd27192e66fa471cf5ee8"
server_key_iv: "60114306f95c6101227f1eb650e3ac9d"
server_key_version: "encryptionkey@0"
I20260812 16:50:54.423667 2143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:54.431531 2143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:54.436685 2143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:54.437026 2143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:54.459532 2143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42521
I20260812 16:50:54.459549 2269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42521 every 8 connection(s)
I20260812 16:50:54.461295 2143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:54.465339 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2143
I20260812 16:50:54.465544 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:54.465942 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:54.466130 29644 external_mini_cluster.cc:1428] Setting key 293d8b27f445f85bb8cc458e5be574c2
I20260812 16:50:54.476238 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:54.468285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:36671 (local address 127.0.0.1:42521)
0812 16:50:54.469863 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:54.469893 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:54.470068 (+ 175us) server_negotiation.cc:407] Connection header received
0812 16:50:54.472083 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:54.472114 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:54.472555 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:54.473037 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:54.474341 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:54.474348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:54.474362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:54.474383 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:54.474451 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:54.474473 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:54.474733 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:54.474953 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 16:50:54.475189 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":779,"threads_started":1}
I20260812 16:50:54.481606 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:54.483526 2271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d: Bootstrap starting.
I20260812 16:50:54.487283 2271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:54.488715 2271 log.cc:824] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:54.491068 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:54.487473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:57093 (local address 127.0.0.1:42521)
0812 16:50:54.487717 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:54.487725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:54.488240 (+ 515us) server_negotiation.cc:407] Connection header received
0812 16:50:54.488490 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:54.488494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:54.488570 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:54.488674 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:54.490145 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:54.490151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:54.490154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:54.490201 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:54.490266 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:54.490291 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:54.490448 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:54.490649 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 16:50:54.490730 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:50:54.493230 2271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d: No bootstrap required, opened a new log
I20260812 16:50:54.502552 2271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124259bc843743b79b575094d916131d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260812 16:50:54.503122 2271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:54.503263 2271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124259bc843743b79b575094d916131d, State: Initialized, Role: FOLLOWER
I20260812 16:50:54.503942 2271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124259bc843743b79b575094d916131d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260812 16:50:54.504190 2271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:54.504302 2271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:54.504436 2271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:54.506763 2271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124259bc843743b79b575094d916131d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260812 16:50:54.507356 2271 leader_election.cc:302] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 124259bc843743b79b575094d916131d; no voters:
I20260812 16:50:54.508061 2271 leader_election.cc:288] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:54.508221 2290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:54.509357 2290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [term 1 LEADER]: Becoming Leader. State: Replica: 124259bc843743b79b575094d916131d, State: Running, Role: LEADER
I20260812 16:50:54.509948 2290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124259bc843743b79b575094d916131d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260812 16:50:54.510507 2271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:54.513324 2294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124259bc843743b79b575094d916131d. Latest consensus state: current_term: 1 leader_uuid: "124259bc843743b79b575094d916131d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124259bc843743b79b575094d916131d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260812 16:50:54.513314 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "124259bc843743b79b575094d916131d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124259bc843743b79b575094d916131d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260812 16:50:54.514216 2294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:54.514310 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:54.524154 2303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:54.530531 2303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:54.544142 2303 catalog_manager.cc:1347] Generated new cluster ID: 30c1e70cdfb043fb9f2a1e6f2471fffa
I20260812 16:50:54.544291 2303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:54.559608 2303 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:54.560149 2303 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:54.575266 2303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 124259bc843743b79b575094d916131d: Generated new TSK 0
I20260812 16:50:54.576254 2303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:54.589903 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2143
W20260812 16:50:54.591511 2263 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:36671
W20260812 16:50:54.592445 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2143: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:54.592641 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2143
2026-08-12T16:50:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:54 dist-test-slave-1vmg krb5kdc[2329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:54 dist-test-slave-1vmg krb5kdc[2329](info): set up 2 sockets
Aug 12 16:50:54 dist-test-slave-1vmg krb5kdc[2329](info): commencing operation
krb5kdc: starting...
W20260812 16:50:55.990617 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.000s sys 0.071s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:56 dist-test-slave-1vmg krb5kdc[2329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:50:56.395568 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:39309
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:39309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:56.832665 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:56.833329 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:56.833458 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:56.841053 2349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:56.841212 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:56.841269 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:56.841337 2349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:56.841382 2349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:56.851897 2349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:39309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:56.853295 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:56.854636 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:56.870988 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:56.871161 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:56.870895 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:56.872020 2349 server_base.cc:1034] running on GCE node
I20260812 16:50:56.873363 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:56.876133 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:56.877540 2349 hybrid_clock.cc:648] HybridClock initialized: now 1786553456877421 us; error 103 us; skew 500 ppm
I20260812 16:50:56.878266 2349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:56 dist-test-slave-1vmg krb5kdc[2329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553456, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:56.882193 2349 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:50:56.883962 2349 webserver.cc:460] Webserver started at http://127.28.243.62:35647/ using document root <none> and password file <none>
I20260812 16:50:56.884712 2349 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:56.884829 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:56.885159 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:56.888622 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "841ac741e195442fb3c38ce1f5722a02"
format_stamp: "Formatted at 2026-08-12 16:50:56 on dist-test-slave-1vmg"
server_key: "64ab117654f9afc0892f66c670953a07"
server_key_iv: "2e1baf0c88d4b17a212e51dbdb4f873f"
server_key_version: "encryptionkey@0"
I20260812 16:50:56.889529 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "841ac741e195442fb3c38ce1f5722a02"
format_stamp: "Formatted at 2026-08-12 16:50:56 on dist-test-slave-1vmg"
server_key: "64ab117654f9afc0892f66c670953a07"
server_key_iv: "2e1baf0c88d4b17a212e51dbdb4f873f"
server_key_version: "encryptionkey@0"
I20260812 16:50:56.898347 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260812 16:50:56.903520 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:56.905215 2349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:50:56.905445 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "841ac741e195442fb3c38ce1f5722a02"
format_stamp: "Formatted at 2026-08-12 16:50:56 on dist-test-slave-1vmg"
server_key: "64ab117654f9afc0892f66c670953a07"
server_key_iv: "2e1baf0c88d4b17a212e51dbdb4f873f"
server_key_version: "encryptionkey@0"
I20260812 16:50:56.905675 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:56.923341 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:56.928149 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:56.928455 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:56.951728 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:39309 every 8 connection(s)
I20260812 16:50:56.951728 2349 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:39309
I20260812 16:50:56.953802 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:56.958583 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:56.960529 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2349
I20260812 16:50:56.960772 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:56.961450 29644 external_mini_cluster.cc:1428] Setting key 4e813b5c7ed385eaa3054cec5abf102d
I20260812 16:50:56.972896 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02: Bootstrap starting.
I20260812 16:50:56.977349 2469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02: Neither blocks nor log segments found. Creating new log.
Aug 12 16:50:56 dist-test-slave-1vmg krb5kdc[2329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:50:56.978482 2469 log.cc:824] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:56.982900 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02: No bootstrap required, opened a new log
I20260812 16:50:56.989243 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:56.963622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48799 (local address 127.28.243.62:39309)
0812 16:50:56.964561 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:56.964584 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:56.964646 (+ 62us) server_negotiation.cc:407] Connection header received
0812 16:50:56.967124 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:56.967159 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:56.967742 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:56.968233 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:56.969820 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:56.971936 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:56.973359 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:56.974965 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:56.975194 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:56.979752 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:56.979773 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:56.979795 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:56.979857 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:56.983925 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:56.984916 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:56.984925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:56.984935 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:56.985186 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:56.985668 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:56.985673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:56.985677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:56.986500 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:56.986844 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:56.987366 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 16:50:56.987761 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":371,"threads_started":1}
I20260812 16:50:56.995613 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:56.995431 2469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841ac741e195442fb3c38ce1f5722a02" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 39309 } }
I20260812 16:50:56.995957 2469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:56.996068 2469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841ac741e195442fb3c38ce1f5722a02, State: Initialized, Role: FOLLOWER
I20260812 16:50:56.996903 2469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841ac741e195442fb3c38ce1f5722a02" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 39309 } }
I20260812 16:50:56.997345 2469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:56.997499 2469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:56.997684 2469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:57.000803 2469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841ac741e195442fb3c38ce1f5722a02" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 39309 } }
I20260812 16:50:57.001536 2469 leader_election.cc:302] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 841ac741e195442fb3c38ce1f5722a02; no voters:
I20260812 16:50:57.002388 2469 leader_election.cc:288] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:57.002526 2486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:57.005338 2469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:57.005506 2486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [term 1 LEADER]: Becoming Leader. State: Replica: 841ac741e195442fb3c38ce1f5722a02, State: Running, Role: LEADER
I20260812 16:50:57.006310 2486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841ac741e195442fb3c38ce1f5722a02" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 39309 } }
I20260812 16:50:57.009984 2488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841ac741e195442fb3c38ce1f5722a02. Latest consensus state: current_term: 1 leader_uuid: "841ac741e195442fb3c38ce1f5722a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841ac741e195442fb3c38ce1f5722a02" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 39309 } } }
I20260812 16:50:57.010475 2488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:57.011126 2487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "841ac741e195442fb3c38ce1f5722a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841ac741e195442fb3c38ce1f5722a02" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 39309 } } }
I20260812 16:50:57.011448 2487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02 [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:57.013037 2504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:57.023785 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:57.001781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39295 (local address 127.28.243.62:39309)
0812 16:50:57.002049 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:57.002055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:57.002355 (+ 300us) server_negotiation.cc:407] Connection header received
0812 16:50:57.002563 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:57.002567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:57.002637 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:57.002735 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:57.005250 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:57.006490 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:57.008630 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:57.009389 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:57.009531 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:57.013604 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:57.013613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:57.013619 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:50:57.013675 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:57.018405 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:57.019610 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:57.019618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:57.019622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:57.020002 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:50:57.020999 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:50:57.021005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:50:57.021008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:50:57.021556 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:57.023106 (+ 1550us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:57.023333 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 16:50:57.023508 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 16:50:57.024740 2504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:57.036973 2504 catalog_manager.cc:1347] Generated new cluster ID: 028e16ddb1d54a76a7d83b003245f3d8
I20260812 16:50:57.037194 2504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:57.078490 2504 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:57.080396 2504 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:57.093392 2504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 841ac741e195442fb3c38ce1f5722a02: Generated new TSK 0
I20260812 16:50:57.096417 2504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:57.114291 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2349
I20260812 16:50:57.116241 2454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:50:57.218221 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2349
W20260812 16:50:57.258721 2327 connection.cc:535] client connection to 127.28.243.62:39309 recv error: Network error: failed to read from TLS socket (remote: 127.28.243.62:39309): Connection reset by peer (error 104)
2026-08-12T16:50:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:57 dist-test-slave-1vmg krb5kdc[2523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:57 dist-test-slave-1vmg krb5kdc[2523](info): set up 2 sockets
Aug 12 16:50:57 dist-test-slave-1vmg krb5kdc[2523](info): commencing operation
krb5kdc: starting...
W20260812 16:50:58.605146 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.001s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:50:58 dist-test-slave-1vmg krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:50:59.014899 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:37529
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:50:59.429117 2549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:50:59.429502 2549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:50:59.429636 2549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:50:59.437583 2549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:50:59.437815 2549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:50:59.437901 2549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:50:59.438016 2549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:50:59.438088 2549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:50:59.449541 2549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:37529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:50:59.451546 2549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:50:59.454059 2549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:50:59.469412 2557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:59.469373 2562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:50:59.469431 2568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:50:59.470038 2549 server_base.cc:1034] running on GCE node
I20260812 16:50:59.471221 2549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:50:59.473382 2549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:50:59.474763 2549 hybrid_clock.cc:648] HybridClock initialized: now 1786553459474699 us; error 48 us; skew 500 ppm
I20260812 16:50:59.475520 2549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:50:59 dist-test-slave-1vmg krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:50:59.484488 2549 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:50:59.486279 2549 webserver.cc:460] Webserver started at http://127.28.243.62:40751/ using document root <none> and password file <none>
I20260812 16:50:59.487149 2549 fs_manager.cc:359] Metadata directory not provided
I20260812 16:50:59.487294 2549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:50:59.487625 2549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:50:59.491314 2549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "72545bef4237483d9b2ecc1b9f363b6f"
format_stamp: "Formatted at 2026-08-12 16:50:59 on dist-test-slave-1vmg"
server_key: "ac2114f1dfd79f0ef02c2f1cc50e4302"
server_key_iv: "37f1cd9443852cc59dbeab084d09c9b3"
server_key_version: "encryptionkey@0"
I20260812 16:50:59.492730 2549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "72545bef4237483d9b2ecc1b9f363b6f"
format_stamp: "Formatted at 2026-08-12 16:50:59 on dist-test-slave-1vmg"
server_key: "ac2114f1dfd79f0ef02c2f1cc50e4302"
server_key_iv: "37f1cd9443852cc59dbeab084d09c9b3"
server_key_version: "encryptionkey@0"
I20260812 16:50:59.501662 2549 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260812 16:50:59.506278 2580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:50:59.508023 2549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:50:59.508239 2549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "72545bef4237483d9b2ecc1b9f363b6f"
format_stamp: "Formatted at 2026-08-12 16:50:59 on dist-test-slave-1vmg"
server_key: "ac2114f1dfd79f0ef02c2f1cc50e4302"
server_key_iv: "37f1cd9443852cc59dbeab084d09c9b3"
server_key_version: "encryptionkey@0"
I20260812 16:50:59.508461 2549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:50:59.526295 2549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:50:59.531522 2549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:50:59.531839 2549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:50:59.557423 2549 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:37529
I20260812 16:50:59.557446 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:37529 every 8 connection(s)
I20260812 16:50:59.559631 2549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:50:59.564777 2683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:50:59.566143 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2549
I20260812 16:50:59.566367 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:50:59.567059 29644 external_mini_cluster.cc:1428] Setting key 860b3edbf5fdb524da060536ef246928
I20260812 16:50:59.583406 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f: Bootstrap starting.
I20260812 16:50:59.586061 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:59.569613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41585 (local address 127.28.243.62:37529)
0812 16:50:59.570653 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:59.570672 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:59.570729 (+ 57us) server_negotiation.cc:407] Connection header received
0812 16:50:59.572726 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:59.572760 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:59.573309 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:59.573741 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:59.575106 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:59.576917 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:59.579507 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:59.580515 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:59.580722 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:59.582587 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:59.582604 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:59.582623 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:59.583424 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:59.583525 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:59.583556 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:59.583750 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:59.584271 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 16:50:59.584602 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":451,"threads_started":1}
I20260812 16:50:59.587589 2683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f: Neither blocks nor log segments found. Creating new log.
I20260812 16:50:59.588837 2683 log.cc:824] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:50:59.591346 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:50:59.593237 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f: No bootstrap required, opened a new log
I20260812 16:50:59.606071 2683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72545bef4237483d9b2ecc1b9f363b6f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37529 } }
I20260812 16:50:59.606757 2683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:50:59.606897 2683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72545bef4237483d9b2ecc1b9f363b6f, State: Initialized, Role: FOLLOWER
I20260812 16:50:59.607010 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:50:59.598216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41047 (local address 127.28.243.62:37529)
0812 16:50:59.598422 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 16:50:59.598428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:50:59.598994 (+ 566us) server_negotiation.cc:407] Connection header received
0812 16:50:59.599144 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:50:59.599149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:50:59.599230 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:50:59.599333 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:50:59.600451 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:59.601408 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:59.603764 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:50:59.604298 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:50:59.604436 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:50:59.605945 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:50:59.605950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:50:59.605954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:50:59.605979 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:50:59.606043 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:50:59.606064 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:50:59.606172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 16:50:59.606536 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 16:50:59.606616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:50:59.607674 2683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72545bef4237483d9b2ecc1b9f363b6f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37529 } }
I20260812 16:50:59.608023 2683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:50:59.608189 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:50:59.608392 2683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:50:59.610855 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72545bef4237483d9b2ecc1b9f363b6f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37529 } }
I20260812 16:50:59.611588 2683 leader_election.cc:302] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72545bef4237483d9b2ecc1b9f363b6f; no voters:
I20260812 16:50:59.612641 2683 leader_election.cc:288] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:50:59.612672 2706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:50:59.614217 2706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [term 1 LEADER]: Becoming Leader. State: Replica: 72545bef4237483d9b2ecc1b9f363b6f, State: Running, Role: LEADER
I20260812 16:50:59.615142 2706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72545bef4237483d9b2ecc1b9f363b6f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37529 } }
I20260812 16:50:59.615948 2683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:50:59.618908 2709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72545bef4237483d9b2ecc1b9f363b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72545bef4237483d9b2ecc1b9f363b6f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37529 } } }
I20260812 16:50:59.619694 2708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72545bef4237483d9b2ecc1b9f363b6f. Latest consensus state: current_term: 1 leader_uuid: "72545bef4237483d9b2ecc1b9f363b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72545bef4237483d9b2ecc1b9f363b6f" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 37529 } } }
I20260812 16:50:59.620056 2708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:59.620085 2709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f [sys.catalog]: This master's current role is: LEADER
I20260812 16:50:59.622121 2714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:50:59.627945 2714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:50:59.642563 2714 catalog_manager.cc:1347] Generated new cluster ID: db2d1a77b86f4eb189197c2ee8d4e4e7
I20260812 16:50:59.642756 2714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:50:59.658205 2714 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:50:59.660485 2714 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:50:59.677131 2714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72545bef4237483d9b2ecc1b9f363b6f: Generated new TSK 0
I20260812 16:50:59.678283 2714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:50:59.699054 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2549
W20260812 16:50:59.701014 2673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41585
W20260812 16:50:59.702289 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:50:59.702504 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2549
W20260812 16:50:59.734382 2520 connection.cc:535] client connection to 127.28.243.62:37529 recv error: Network error: failed to read from TLS socket (remote: 127.28.243.62:37529): Connection reset by peer (error 104)
2026-08-12T16:50:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:50:59 dist-test-slave-1vmg krb5kdc[2743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:50:59 dist-test-slave-1vmg krb5kdc[2743](info): set up 2 sockets
Aug 12 16:50:59 dist-test-slave-1vmg krb5kdc[2743](info): commencing operation
krb5kdc: starting...
W20260812 16:51:01.076270 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.008s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:51:01 dist-test-slave-1vmg krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:51:01.499920 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45687
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:45687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:51:01.946993 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:01.947351 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:01.947479 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:01.956261 2770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:01.956414 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:01.956463 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:01.956534 2770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:01.956611 2770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:01.968355 2770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:45687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:01.969780 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:01.972002 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:01.986835 2787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:01.986958 2785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:01.987226 2783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:01.987769 2770 server_base.cc:1034] running on GCE node
I20260812 16:51:01.988612 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:01.990295 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:01.991616 2770 hybrid_clock.cc:648] HybridClock initialized: now 1786553461991535 us; error 44 us; skew 500 ppm
I20260812 16:51:01.992213 2770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:01 dist-test-slave-1vmg krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553461, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:01.996960 2770 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:51:01.999779 2770 webserver.cc:460] Webserver started at http://127.28.243.62:41671/ using document root <none> and password file <none>
I20260812 16:51:02.000872 2770 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:02.001029 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:02.001519 2770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:02.005225 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "8b95e06a59d548ab8654b014137a7787"
format_stamp: "Formatted at 2026-08-12 16:51:01 on dist-test-slave-1vmg"
server_key: "fdf900102c25bea96b9949fd5e00ee78"
server_key_iv: "3800f42d946e72b7bdde5b7eeb3b7ec5"
server_key_version: "encryptionkey@0"
I20260812 16:51:02.006126 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "8b95e06a59d548ab8654b014137a7787"
format_stamp: "Formatted at 2026-08-12 16:51:01 on dist-test-slave-1vmg"
server_key: "fdf900102c25bea96b9949fd5e00ee78"
server_key_iv: "3800f42d946e72b7bdde5b7eeb3b7ec5"
server_key_version: "encryptionkey@0"
I20260812 16:51:02.013548 2770 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 16:51:02.019284 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:02.021046 2770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 16:51:02.021273 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "8b95e06a59d548ab8654b014137a7787"
format_stamp: "Formatted at 2026-08-12 16:51:01 on dist-test-slave-1vmg"
server_key: "fdf900102c25bea96b9949fd5e00ee78"
server_key_iv: "3800f42d946e72b7bdde5b7eeb3b7ec5"
server_key_version: "encryptionkey@0"
I20260812 16:51:02.021507 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:02.028414 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:02.037876 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:02.038489 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:02.062588 2770 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:45687
I20260812 16:51:02.062620 2901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:45687 every 8 connection(s)
I20260812 16:51:02.064307 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:51:02.068758 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:02.073715 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 2770
I20260812 16:51:02.073944 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:51:02.074491 29644 external_mini_cluster.cc:1428] Setting key d7d32a3a060f948341b363d7742ac452
I20260812 16:51:02.084966 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:02.077145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42293 (local address 127.28.243.62:45687)
0812 16:51:02.078117 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:02.078132 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:02.078184 (+ 52us) server_negotiation.cc:407] Connection header received
0812 16:51:02.080215 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:02.080249 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:02.081046 (+ 797us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:02.081548 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:02.083138 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:02.083146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:02.083163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:51:02.083202 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:02.083278 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:51:02.083303 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:02.083511 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:02.083630 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 16:51:02.083872 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":400,"threads_started":1}
I20260812 16:51:02.085218 2904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787: Bootstrap starting.
I20260812 16:51:02.088814 2904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:02.089174 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:51:02.090046 2904 log.cc:824] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:02.095130 2904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787: No bootstrap required, opened a new log
I20260812 16:51:02.099542 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:02.095886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49581 (local address 127.28.243.62:45687)
0812 16:51:02.096406 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:02.096414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:02.096658 (+ 244us) server_negotiation.cc:407] Connection header received
0812 16:51:02.096890 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:02.096894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:02.096962 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:02.097085 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:02.098774 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:02.098779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:02.098783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:51:02.098801 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:02.098850 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:51:02.098867 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:02.099061 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:02.099174 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 16:51:02.099232 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260812 16:51:02.106096 2904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b95e06a59d548ab8654b014137a7787" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45687 } }
I20260812 16:51:02.106583 2904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:51:02.106686 2904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b95e06a59d548ab8654b014137a7787, State: Initialized, Role: FOLLOWER
I20260812 16:51:02.107437 2904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b95e06a59d548ab8654b014137a7787" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45687 } }
I20260812 16:51:02.107761 2904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:51:02.107916 2904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:51:02.108132 2904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:02.110620 2904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b95e06a59d548ab8654b014137a7787" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45687 } }
I20260812 16:51:02.111384 2904 leader_election.cc:302] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b95e06a59d548ab8654b014137a7787; no voters:
I20260812 16:51:02.112025 2904 leader_election.cc:288] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:51:02.112193 2921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:51:02.113495 2921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [term 1 LEADER]: Becoming Leader. State: Replica: 8b95e06a59d548ab8654b014137a7787, State: Running, Role: LEADER
I20260812 16:51:02.114403 2921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b95e06a59d548ab8654b014137a7787" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45687 } }
I20260812 16:51:02.114724 2904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:02.120609 2925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b95e06a59d548ab8654b014137a7787. Latest consensus state: current_term: 1 leader_uuid: "8b95e06a59d548ab8654b014137a7787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b95e06a59d548ab8654b014137a7787" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45687 } } }
I20260812 16:51:02.121387 2925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:02.121387 2922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b95e06a59d548ab8654b014137a7787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b95e06a59d548ab8654b014137a7787" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45687 } } }
I20260812 16:51:02.121721 2922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:02.124406 2939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:51:02.129865 2939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:51:02.142299 2939 catalog_manager.cc:1347] Generated new cluster ID: d05f31a6b92f4dbeb6fd6e053fd1188f
I20260812 16:51:02.142441 2939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:51:02.162689 2939 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:51:02.163249 2939 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:51:02.172873 2939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b95e06a59d548ab8654b014137a7787: Generated new TSK 0
I20260812 16:51:02.174551 2939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:51:02.194128 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2770
W20260812 16:51:02.195926 2890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42293
W20260812 16:51:02.196803 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 2770: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:51:02.197037 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 2770
2026-08-12T16:51:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:51:02 dist-test-slave-1vmg krb5kdc[2978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:51:02 dist-test-slave-1vmg krb5kdc[2978](info): set up 2 sockets
Aug 12 16:51:02 dist-test-slave-1vmg krb5kdc[2978](info): commencing operation
krb5kdc: starting...
W20260812 16:51:03.631703 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.378s user 0.000s sys 0.071s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:51:03 dist-test-slave-1vmg krb5kdc[2978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:51:04.044564 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46779
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:51:04.469367 3002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:04.469692 3002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:04.469789 3002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:04.477561 3002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:04.477726 3002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:04.477778 3002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:04.477833 3002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:04.477869 3002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:04.490585 3002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:04.492781 3002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:04.494625 3002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:04.508489 3011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:04.508556 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:04.508567 3008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:04.509222 3002 server_base.cc:1034] running on GCE node
I20260812 16:51:04.510327 3002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:04.512296 3002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:04.513593 3002 hybrid_clock.cc:648] HybridClock initialized: now 1786553464513548 us; error 47 us; skew 500 ppm
I20260812 16:51:04.514369 3002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:04 dist-test-slave-1vmg krb5kdc[2978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553464, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:04.526485 3002 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:51:04.528399 3002 webserver.cc:460] Webserver started at http://127.28.243.62:43411/ using document root <none> and password file <none>
I20260812 16:51:04.529342 3002 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:04.529492 3002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:04.529825 3002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:04.533273 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "931819ca653a455f8f9e3cf09d24608a"
format_stamp: "Formatted at 2026-08-12 16:51:04 on dist-test-slave-1vmg"
server_key: "6272e514439cab7c042d768dc23acf16"
server_key_iv: "d615c43d88ba9f37421df1c1a79ff9fb"
server_key_version: "encryptionkey@0"
I20260812 16:51:04.534109 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "931819ca653a455f8f9e3cf09d24608a"
format_stamp: "Formatted at 2026-08-12 16:51:04 on dist-test-slave-1vmg"
server_key: "6272e514439cab7c042d768dc23acf16"
server_key_iv: "d615c43d88ba9f37421df1c1a79ff9fb"
server_key_version: "encryptionkey@0"
I20260812 16:51:04.539973 3002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 16:51:04.543992 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:04.546061 3002 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:51:04.546377 3002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "931819ca653a455f8f9e3cf09d24608a"
format_stamp: "Formatted at 2026-08-12 16:51:04 on dist-test-slave-1vmg"
server_key: "6272e514439cab7c042d768dc23acf16"
server_key_iv: "d615c43d88ba9f37421df1c1a79ff9fb"
server_key_version: "encryptionkey@0"
I20260812 16:51:04.546670 3002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:04.562048 3002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:04.566875 3002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:04.567384 3002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:04.588967 3002 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:46779
I20260812 16:51:04.589254 3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:46779 every 8 connection(s)
I20260812 16:51:04.590590 3002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:51:04.590895 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3002
I20260812 16:51:04.591116 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:51:04.591912 29644 external_mini_cluster.cc:1428] Setting key 4858cf3e69b681562e075ca7e810e53c
I20260812 16:51:04.597873 3127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 16:51:04 dist-test-slave-1vmg krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:51:04.613153 3127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a: Bootstrap starting.
I20260812 16:51:04.619005 3127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:04.620837 3127 log.cc:824] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:04.621171 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:04.594202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51321 (local address 127.28.243.62:46779)
0812 16:51:04.595381 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:04.595394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:04.595440 (+ 46us) server_negotiation.cc:407] Connection header received
0812 16:51:04.597377 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:04.597404 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:04.597873 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:04.598285 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:04.600318 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:04.601772 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:04.603737 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:04.604564 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:04.604719 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:04.610635 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:04.610653 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:04.610679 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:04.610752 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:04.614523 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:04.616208 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:04.616217 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:04.616226 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:04.616477 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:04.617374 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:04.617379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:04.617382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:04.618152 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:04.618405 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:04.619006 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 16:51:04.619446 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":444,"threads_started":1}
I20260812 16:51:04.624504 3127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a: No bootstrap required, opened a new log
I20260812 16:51:04.626097 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:51:04.634935 3127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931819ca653a455f8f9e3cf09d24608a" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46779 } }
I20260812 16:51:04.635496 3127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:51:04.635782 3127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931819ca653a455f8f9e3cf09d24608a, State: Initialized, Role: FOLLOWER
I20260812 16:51:04.636943 3127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931819ca653a455f8f9e3cf09d24608a" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46779 } }
I20260812 16:51:04.637359 3127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:51:04.637560 3127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:51:04.637777 3127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:04.640960 3127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931819ca653a455f8f9e3cf09d24608a" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46779 } }
I20260812 16:51:04.641767 3127 leader_election.cc:302] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931819ca653a455f8f9e3cf09d24608a; no voters:
I20260812 16:51:04.642511 3127 leader_election.cc:288] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:51:04.642642 3146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:51:04.645264 3127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:04.646481 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:04.631802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57711 (local address 127.28.243.62:46779)
0812 16:51:04.632111 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:04.632118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:04.632340 (+ 222us) server_negotiation.cc:407] Connection header received
0812 16:51:04.632482 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:04.632487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:04.632580 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:04.632717 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:04.634158 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:04.635258 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:04.636614 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:04.637135 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:04.637259 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:04.639817 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:04.639823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:04.639827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:04.639867 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:04.643108 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:04.643976 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:04.643980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:04.643982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:04.644134 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:04.644720 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:04.644726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:04.644729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:04.645060 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:04.645444 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:04.645795 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 16:51:04.646063 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:51:04.649598 3146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [term 1 LEADER]: Becoming Leader. State: Replica: 931819ca653a455f8f9e3cf09d24608a, State: Running, Role: LEADER
I20260812 16:51:04.650892 3146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931819ca653a455f8f9e3cf09d24608a" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46779 } }
I20260812 16:51:04.656453 3147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931819ca653a455f8f9e3cf09d24608a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931819ca653a455f8f9e3cf09d24608a" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46779 } } }
I20260812 16:51:04.656917 3147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:04.659803 3148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931819ca653a455f8f9e3cf09d24608a. Latest consensus state: current_term: 1 leader_uuid: "931819ca653a455f8f9e3cf09d24608a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931819ca653a455f8f9e3cf09d24608a" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46779 } } }
I20260812 16:51:04.660604 3148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a [sys.catalog]: This master's current role is: LEADER
W20260812 16:51:04.667848 3176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:51:04.668022 3176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:51:04.668185 3165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:51:04.670121 3165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:51:04.681757 3165 catalog_manager.cc:1347] Generated new cluster ID: 084981a8acc34520955eb16c66aeb29f
I20260812 16:51:04.681903 3165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:51:04.704481 3165 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:51:04.706038 3165 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:51:04.726231 3165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931819ca653a455f8f9e3cf09d24608a: Generated new TSK 0
I20260812 16:51:04.727208 3165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:51:04.747040 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3002
I20260812 16:51:04.748342 3120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:04.861256 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3002
2026-08-12T16:51:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2662 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:51:05 dist-test-slave-1vmg krb5kdc[3184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:51:05 dist-test-slave-1vmg krb5kdc[3184](info): set up 2 sockets
Aug 12 16:51:05 dist-test-slave-1vmg krb5kdc[3184](info): commencing operation
krb5kdc: starting...
W20260812 16:51:06.248549 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.011s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:51:06 dist-test-slave-1vmg krb5kdc[3184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:51:06.618446 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38753
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:38753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:51:07.005414 3216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:07.006028 3216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:07.006410 3216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:07.015554 3216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:07.015717 3216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:07.015796 3216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:07.015895 3216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:07.015954 3216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:07.027034 3216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:38753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:38753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:07.028540 3216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:07.030002 3216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:07.045565 3231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:07.045593 3229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:07.045755 3227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:07.047346 3216 server_base.cc:1034] running on GCE node
I20260812 16:51:07.048295 3216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:07.050004 3216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:07.051268 3216 hybrid_clock.cc:648] HybridClock initialized: now 1786553467051222 us; error 55 us; skew 500 ppm
I20260812 16:51:07.051680 3216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:07 dist-test-slave-1vmg krb5kdc[3184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553467, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:07.055279 3216 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:51:07.056945 3216 webserver.cc:460] Webserver started at http://127.28.243.62:37689/ using document root <none> and password file <none>
I20260812 16:51:07.057695 3216 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:07.057828 3216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:07.058174 3216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:07.061570 3216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "92717a227d064dbb9e1f90ac5817a522"
format_stamp: "Formatted at 2026-08-12 16:51:07 on dist-test-slave-1vmg"
server_key: "dc8baff1c1e50ff20f29da5dd18024cb"
server_key_iv: "a434a6670f8fb788e57f99dd64166491"
server_key_version: "encryptionkey@0"
I20260812 16:51:07.062415 3216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "92717a227d064dbb9e1f90ac5817a522"
format_stamp: "Formatted at 2026-08-12 16:51:07 on dist-test-slave-1vmg"
server_key: "dc8baff1c1e50ff20f29da5dd18024cb"
server_key_iv: "a434a6670f8fb788e57f99dd64166491"
server_key_version: "encryptionkey@0"
I20260812 16:51:07.068339 3216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.001s
I20260812 16:51:07.072540 3240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:07.074339 3216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:51:07.074550 3216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "92717a227d064dbb9e1f90ac5817a522"
format_stamp: "Formatted at 2026-08-12 16:51:07 on dist-test-slave-1vmg"
server_key: "dc8baff1c1e50ff20f29da5dd18024cb"
server_key_iv: "a434a6670f8fb788e57f99dd64166491"
server_key_version: "encryptionkey@0"
I20260812 16:51:07.074769 3216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:07.092193 3216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:07.097565 3216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:07.097886 3216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:07.121596 3216 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:38753
I20260812 16:51:07.121706 3326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:38753 every 8 connection(s)
I20260812 16:51:07.123281 3216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:51:07.127313 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3216
I20260812 16:51:07.127653 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:51:07.128191 29644 external_mini_cluster.cc:1428] Setting key f6a185dbebcf25d82503f077fbaa0ee1
I20260812 16:51:07.128430 3327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:07.145648 3327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522: Bootstrap starting.
I20260812 16:51:07.146008 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:07.130222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57541 (local address 127.28.243.62:38753)
0812 16:51:07.131114 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:07.131130 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:07.131184 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:51:07.133889 (+ 2705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:07.133921 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:07.134531 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:07.135074 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:07.135675 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:07.137727 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:07.139826 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:07.140953 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:07.141182 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:07.142819 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:07.142907 (+ 88us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:07.142974 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:51:07.143750 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:07.143863 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:51:07.143904 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:07.144037 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:07.144304 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 16:51:07.144606 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":359,"threads_started":1}
I20260812 16:51:07.149713 3327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:07.150555 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:51:07.151113 3327 log.cc:824] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:07.154852 3327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522: No bootstrap required, opened a new log
I20260812 16:51:07.163947 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:07.155781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45657 (local address 127.28.243.62:38753)
0812 16:51:07.155970 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:07.155976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:07.156448 (+ 472us) server_negotiation.cc:407] Connection header received
0812 16:51:07.156606 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:07.156612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:07.156707 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:07.156838 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:07.157480 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:07.158610 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:07.160062 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:07.161140 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:07.161410 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:07.162903 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:07.162940 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:07.162945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:51:07.162978 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:07.163043 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:51:07.163066 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:07.163192 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:07.163423 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 16:51:07.163566 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:51:07.165645 3327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92717a227d064dbb9e1f90ac5817a522" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38753 } }
I20260812 16:51:07.166088 3327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:51:07.166185 3327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92717a227d064dbb9e1f90ac5817a522, State: Initialized, Role: FOLLOWER
I20260812 16:51:07.166783 3327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92717a227d064dbb9e1f90ac5817a522" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38753 } }
I20260812 16:51:07.167078 3327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:51:07.167213 3327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:51:07.167387 3327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:07.169723 3327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92717a227d064dbb9e1f90ac5817a522" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38753 } }
I20260812 16:51:07.170331 3327 leader_election.cc:302] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92717a227d064dbb9e1f90ac5817a522; no voters:
I20260812 16:51:07.171227 3327 leader_election.cc:288] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:51:07.171365 3345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:51:07.172521 3345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [term 1 LEADER]: Becoming Leader. State: Replica: 92717a227d064dbb9e1f90ac5817a522, State: Running, Role: LEADER
I20260812 16:51:07.173144 3345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92717a227d064dbb9e1f90ac5817a522" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38753 } }
I20260812 16:51:07.173722 3327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:07.176199 3346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92717a227d064dbb9e1f90ac5817a522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92717a227d064dbb9e1f90ac5817a522" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38753 } } }
I20260812 16:51:07.176694 3346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:07.176229 3350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92717a227d064dbb9e1f90ac5817a522. Latest consensus state: current_term: 1 leader_uuid: "92717a227d064dbb9e1f90ac5817a522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92717a227d064dbb9e1f90ac5817a522" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 38753 } } }
I20260812 16:51:07.177335 3350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:07.179255 3360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:51:07.185360 3360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:51:07.196957 3360 catalog_manager.cc:1347] Generated new cluster ID: 6e7b1398d67240aaa201275a8cfc8753
I20260812 16:51:07.197093 3360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:51:07.250247 3360 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:51:07.251897 3360 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:51:07.266137 3360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92717a227d064dbb9e1f90ac5817a522: Generated new TSK 0
I20260812 16:51:07.267818 3360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:51:07.280701 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3216
W20260812 16:51:07.282702 3316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57541
W20260812 16:51:07.283641 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:51:07.283811 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3216
2026-08-12T16:51:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:51:07 dist-test-slave-1vmg krb5kdc[3386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:51:07 dist-test-slave-1vmg krb5kdc[3386](info): set up 2 sockets
Aug 12 16:51:07 dist-test-slave-1vmg krb5kdc[3386](info): commencing operation
krb5kdc: starting...
W20260812 16:51:08.684194 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:51:08 dist-test-slave-1vmg krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:51:09.087095 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45163
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:45163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:51:09.515565 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:09.515908 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:09.516004 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:09.524174 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:09.524391 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:09.524461 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:09.524567 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:09.524631 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:09.536619 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:45163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:45163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:09.538116 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:09.539623 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:09.554044 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:09.554077 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:09.554143 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:09.555269 3409 server_base.cc:1034] running on GCE node
I20260812 16:51:09.556172 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:09.557900 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:09.559137 3409 hybrid_clock.cc:648] HybridClock initialized: now 1786553469559090 us; error 50 us; skew 500 ppm
I20260812 16:51:09.559554 3409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:09 dist-test-slave-1vmg krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553469, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:09.569357 3409 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:51:09.571719 3409 webserver.cc:460] Webserver started at http://127.28.243.62:42027/ using document root <none> and password file <none>
I20260812 16:51:09.572757 3409 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:09.572918 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:09.573395 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:09.577795 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "4a09576fec2241659890c69c6d339a35"
format_stamp: "Formatted at 2026-08-12 16:51:09 on dist-test-slave-1vmg"
server_key: "2d363ed45e1758c6b45a2ee0df846174"
server_key_iv: "f687be983cacfd05fd541f4444fb0522"
server_key_version: "encryptionkey@0"
I20260812 16:51:09.578962 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "4a09576fec2241659890c69c6d339a35"
format_stamp: "Formatted at 2026-08-12 16:51:09 on dist-test-slave-1vmg"
server_key: "2d363ed45e1758c6b45a2ee0df846174"
server_key_iv: "f687be983cacfd05fd541f4444fb0522"
server_key_version: "encryptionkey@0"
I20260812 16:51:09.586440 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260812 16:51:09.591017 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:09.592984 3409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:51:09.593214 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "4a09576fec2241659890c69c6d339a35"
format_stamp: "Formatted at 2026-08-12 16:51:09 on dist-test-slave-1vmg"
server_key: "2d363ed45e1758c6b45a2ee0df846174"
server_key_iv: "f687be983cacfd05fd541f4444fb0522"
server_key_version: "encryptionkey@0"
I20260812 16:51:09.593459 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:09.600435 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:09.605751 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:09.606093 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:09.627187 3409 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:45163
I20260812 16:51:09.627235 3546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:45163 every 8 connection(s)
I20260812 16:51:09.628814 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:51:09.629675 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3409
I20260812 16:51:09.629868 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:51:09.630388 29644 external_mini_cluster.cc:1428] Setting key 071c14fe743d72ec9e7004caf5ae4b5e
I20260812 16:51:09.634171 3548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:09.641599 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:09.633572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37363 (local address 127.28.243.62:45163)
0812 16:51:09.634951 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:09.634973 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:09.635034 (+ 61us) server_negotiation.cc:407] Connection header received
0812 16:51:09.637091 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:09.637121 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:09.637630 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:09.638090 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:09.639716 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:09.639725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:09.639746 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:51:09.639786 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:09.639885 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:51:09.639915 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:09.640146 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:09.640262 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 16:51:09.640522 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":548,"threads_started":1}
I20260812 16:51:09.647504 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:51:09.648931 3548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35: Bootstrap starting.
I20260812 16:51:09.653033 3548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:09.654409 3548 log.cc:824] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:09.656316 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:09.652673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40641 (local address 127.28.243.62:45163)
0812 16:51:09.652910 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:09.652917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:09.653663 (+ 746us) server_negotiation.cc:407] Connection header received
0812 16:51:09.653908 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:09.653914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:09.653990 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:09.654155 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:09.655476 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:09.655481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:09.655484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:51:09.655507 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:09.655629 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:51:09.655656 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:09.655776 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:09.655956 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 16:51:09.656016 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:51:09.659451 3548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35: No bootstrap required, opened a new log
I20260812 16:51:09.668040 3548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a09576fec2241659890c69c6d339a35" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45163 } }
I20260812 16:51:09.668480 3548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:51:09.668577 3548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a09576fec2241659890c69c6d339a35, State: Initialized, Role: FOLLOWER
I20260812 16:51:09.669273 3548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a09576fec2241659890c69c6d339a35" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45163 } }
I20260812 16:51:09.669552 3548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:51:09.669723 3548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:51:09.669925 3548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:09.672380 3548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a09576fec2241659890c69c6d339a35" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45163 } }
I20260812 16:51:09.672972 3548 leader_election.cc:302] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a09576fec2241659890c69c6d339a35; no voters:
I20260812 16:51:09.673573 3548 leader_election.cc:288] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:51:09.673730 3568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:51:09.674597 3568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [term 1 LEADER]: Becoming Leader. State: Replica: 4a09576fec2241659890c69c6d339a35, State: Running, Role: LEADER
I20260812 16:51:09.675420 3568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a09576fec2241659890c69c6d339a35" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45163 } }
I20260812 16:51:09.676540 3548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:09.678486 3572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a09576fec2241659890c69c6d339a35. Latest consensus state: current_term: 1 leader_uuid: "4a09576fec2241659890c69c6d339a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a09576fec2241659890c69c6d339a35" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45163 } } }
I20260812 16:51:09.678519 3569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a09576fec2241659890c69c6d339a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a09576fec2241659890c69c6d339a35" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 45163 } } }
I20260812 16:51:09.679025 3569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:09.679031 3572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:09.680137 3583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:51:09.686697 3583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:51:09.697746 3583 catalog_manager.cc:1347] Generated new cluster ID: 818dc587c1974c2892f93ce4acec9f2e
I20260812 16:51:09.697919 3583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:51:09.717548 3583 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:51:09.718511 3583 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:51:09.732950 3583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a09576fec2241659890c69c6d339a35: Generated new TSK 0
I20260812 16:51:09.734740 3583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:51:09.742970 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3409
W20260812 16:51:09.744330 3538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37363
W20260812 16:51:09.745085 29644 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3409: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:51:09.745268 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3409
2026-08-12T16:51:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2461 ms)
[----------] 18 tests from AuthTokenIssuingTest (45540 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:51:09 dist-test-slave-1vmg krb5kdc[3609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:51:09 dist-test-slave-1vmg krb5kdc[3609](info): set up 2 sockets
Aug 12 16:51:09 dist-test-slave-1vmg krb5kdc[3609](info): commencing operation
krb5kdc: starting...
W20260812 16:51:11.112912 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.001s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:51:11 dist-test-slave-1vmg krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:51:11.515221 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43935
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43935,127.28.243.61:37921,127.28.243.60:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:51:11.926367 3635 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:51:11.926743 3635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:11.926856 3635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:11.927011 3635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:11.934888 3635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:11.935024 3635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:11.935101 3635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:11.935179 3635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:11.935302 3635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:11.946404 3635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43935,127.28.243.61:37921,127.28.243.60:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:43935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:11.947956 3635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:11.949417 3635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:11.963143 3646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:11.963224 3643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:11.963208 3644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:11.964113 3635 server_base.cc:1034] running on GCE node
I20260812 16:51:11.965018 3635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:11.966615 3635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:11.967861 3635 hybrid_clock.cc:648] HybridClock initialized: now 1786553471967760 us; error 78 us; skew 500 ppm
I20260812 16:51:11.968254 3635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:11 dist-test-slave-1vmg krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:11.971921 3635 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:51:11.973928 3635 webserver.cc:460] Webserver started at http://127.28.243.62:40693/ using document root <none> and password file <none>
I20260812 16:51:11.974726 3635 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:11.974839 3635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:11.975238 3635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:11.978571 3635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "800b0de8e9b743eb9512b813154f8c15"
format_stamp: "Formatted at 2026-08-12 16:51:11 on dist-test-slave-1vmg"
server_key: "c12e38252436fd3dd2c3b562482d14aa"
server_key_iv: "74038c06c8fcb0292a1155b83662b157"
server_key_version: "encryptionkey@0"
I20260812 16:51:11.979502 3635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "800b0de8e9b743eb9512b813154f8c15"
format_stamp: "Formatted at 2026-08-12 16:51:11 on dist-test-slave-1vmg"
server_key: "c12e38252436fd3dd2c3b562482d14aa"
server_key_iv: "74038c06c8fcb0292a1155b83662b157"
server_key_version: "encryptionkey@0"
I20260812 16:51:11.985445 3635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:51:11.989871 3656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:11.992008 3635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:51:11.992259 3635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "800b0de8e9b743eb9512b813154f8c15"
format_stamp: "Formatted at 2026-08-12 16:51:11 on dist-test-slave-1vmg"
server_key: "c12e38252436fd3dd2c3b562482d14aa"
server_key_iv: "74038c06c8fcb0292a1155b83662b157"
server_key_version: "encryptionkey@0"
I20260812 16:51:11.992522 3635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:12.011250 3635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:12.016110 3635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:12.016404 3635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:12.040740 3635 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:43935
I20260812 16:51:12.040776 3757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:43935 every 8 connection(s)
I20260812 16:51:12.042276 3635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:51:12.045864 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3635
I20260812 16:51:12.046128 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:51:12.047024 29644 external_mini_cluster.cc:1428] Setting key eb04120f0e1cd717f8e99f4862073e80
I20260812 16:51:12.047307 3758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:12.058585 3758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:12 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.61@KRBTEST.COM" created.
I20260812 16:51:12.091895 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.064580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43935 (local address 127.0.0.1:56066)
0812 16:51:12.065543 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.065602 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:12.067947 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:12.069210 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:12.069216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:12.069848 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:12.070224 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.070244 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.072699 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.072704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.074653 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.074690 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.076850 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.076866 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.077224 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.078436 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:12.078502 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:12.081739 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:12.086888 (+ 5149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.086902 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.086973 (+ 71us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.087472 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.088059 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.088063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.088066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.088255 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.089443 (+ 1188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:12.089451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:12.089688 (+ 237us) client_negotiation.cc:769] Sending connection context
0812 16:51:12.090026 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 16:51:12.090792 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":235,"thread_start_us":343,"threads_started":1}
I20260812 16:51:12.091933 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.065854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.28.243.62:43935)
0812 16:51:12.066496 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:12.066502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:12.066550 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:51:12.068473 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:12.068483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:12.068866 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:12.069120 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:12.070475 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.072433 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.075007 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.076043 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.076305 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.082203 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:12.082228 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:12.082251 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:12.082312 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:12.086632 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.087646 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.087657 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.087661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.087882 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.088383 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.088387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.088389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.089239 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:12.089426 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:12.090139 (+ 713us) server_negotiation.cc:299] Negotiation successful
0812 16:51:12.090582 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":458,"thread_start_us":306,"threads_started":1}
I20260812 16:51:12.096396 3758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:51:12.097903 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.097343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38648)
0812 16:51:12.097580 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.097734 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260812 16:51:12.098268 3657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
I20260812 16:51:12.100374 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.099571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38654)
0812 16:51:12.100084 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.100188 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
W20260812 16:51:12.100643 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37921: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab.
Entry for principal kudu/127.28.243.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab.
I20260812 16:51:12.141181 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:37921
--webserver_interface=127.28.243.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab
--principal=kudu/127.28.243.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43935,127.28.243.61:37921,127.28.243.60:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:51:12.150133 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } attempt: 1
I20260812 16:51:12.152274 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.151194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38668)
0812 16:51:12.151673 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.151909 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260812 16:51:12.154157 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.153461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38678)
0812 16:51:12.153791 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.153930 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260812 16:51:12.154455 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37921: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
I20260812 16:51:12.222777 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } attempt: 2
I20260812 16:51:12.224073 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.223536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38690)
0812 16:51:12.223808 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.223914 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 16:51:12.225292 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.224846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38696)
0812 16:51:12.225039 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.225146 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 16:51:12.225508 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37921: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
I20260812 16:51:12.316797 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } attempt: 3
I20260812 16:51:12.318327 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.317625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38710)
0812 16:51:12.317846 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.318068 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 16:51:12.320250 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.319414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38712)
0812 16:51:12.319742 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.320043 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260812 16:51:12.320636 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37921: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
I20260812 16:51:12.463968 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } attempt: 4
I20260812 16:51:12.465269 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.464684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38714)
0812 16:51:12.465001 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.465095 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260812 16:51:12.466802 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.466366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38716)
0812 16:51:12.466547 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.466636 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 16:51:12.467036 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:37921: Network error: Client connection negotiation failed: client connection to 127.28.243.61:37921: connect: Connection refused (error 111)
W20260812 16:51:12.532507 3768 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:51:12.533036 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:12.533138 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:12.533259 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:12.543502 3768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:12.543651 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:12.543718 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:12.543784 3768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:12.543887 3768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:12.557927 3768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43935,127.28.243.61:37921,127.28.243.60:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab
--principal=kudu/127.28.243.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:12.560227 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:12.561717 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:12.577941 3776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:12.578219 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:12.578219 3775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:12.579071 3768 server_base.cc:1034] running on GCE node
I20260812 16:51:12.580063 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:12.581736 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:12.583149 3768 hybrid_clock.cc:648] HybridClock initialized: now 1786553472583062 us; error 147 us; skew 500 ppm
I20260812 16:51:12.583552 3768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:12 dist-test-slave-1vmg krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:12.587659 3768 init.cc:377] Logged in from keytab as kudu/127.28.243.61@KRBTEST.COM (short username kudu)
I20260812 16:51:12.589632 3768 webserver.cc:460] Webserver started at http://127.28.243.61:39127/ using document root <none> and password file <none>
I20260812 16:51:12.590433 3768 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:12.590561 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:12.590899 3768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:12.595373 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data/instance:
uuid: "96bbfd666bd242739c32c9e45c335e87"
format_stamp: "Formatted at 2026-08-12 16:51:12 on dist-test-slave-1vmg"
server_key: "3f02d2c78b20268f316305bc524183b2"
server_key_iv: "0e9933bf801887027ff970f0c5de0dac"
server_key_version: "encryptionkey@0"
I20260812 16:51:12.596673 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal/instance:
uuid: "96bbfd666bd242739c32c9e45c335e87"
format_stamp: "Formatted at 2026-08-12 16:51:12 on dist-test-slave-1vmg"
server_key: "3f02d2c78b20268f316305bc524183b2"
server_key_iv: "0e9933bf801887027ff970f0c5de0dac"
server_key_version: "encryptionkey@0"
I20260812 16:51:12.604580 3768 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260812 16:51:12.609915 3792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:12.611753 3768 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:51:12.612124 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal
uuid: "96bbfd666bd242739c32c9e45c335e87"
format_stamp: "Formatted at 2026-08-12 16:51:12 on dist-test-slave-1vmg"
server_key: "3f02d2c78b20268f316305bc524183b2"
server_key_iv: "0e9933bf801887027ff970f0c5de0dac"
server_key_version: "encryptionkey@0"
I20260812 16:51:12.612427 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:12.636883 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:12.642761 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:12.643118 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:12.666517 3768 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.61:37921
I20260812 16:51:12.666527 3885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.61:37921 every 8 connection(s)
I20260812 16:51:12.668021 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
I20260812 16:51:12.669781 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3768
I20260812 16:51:12.670011 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-1/wal/instance
I20260812 16:51:12.670581 29644 external_mini_cluster.cc:1428] Setting key 1528f8eda10a0ca51b492f96786ba998
I20260812 16:51:12.673759 3886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:12.683370 3886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:12 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:51:12.710875 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.687006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.28.243.62:43935)
0812 16:51:12.687604 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:12.687610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:12.688771 (+ 1161us) server_negotiation.cc:407] Connection header received
0812 16:51:12.691228 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:12.691238 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:12.691405 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:12.691614 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:12.693703 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.695019 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.697120 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.698011 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.698178 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.702938 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:12.702948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:12.702952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:12.703008 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:12.706687 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.707715 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.707721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.707724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.707880 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.708716 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.708721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.708725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.709130 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:12.709302 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:12.710185 (+ 883us) server_negotiation.cc:299] Negotiation successful
0812 16:51:12.710460 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":298,"threads_started":1}
I20260812 16:51:12.711715 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.687630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43935 (local address 127.0.0.1:56070)
0812 16:51:12.688534 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.688602 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:12.690681 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:12.691931 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:12.691946 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:12.692679 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:12.693370 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.693391 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.695360 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.695365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.696883 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.696905 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.698297 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.698303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.698540 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.699351 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:12.699378 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:12.702565 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:12.706977 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.706991 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.707014 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.707565 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.708127 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.708134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.708137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.708548 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.709354 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:12.709362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:12.709765 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 16:51:12.710021 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 16:51:12.710475 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.60@KRBTEST.COM; defaulting to no policy
I20260812 16:51:12.714133 3886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.60@KRBTEST.COM" created.
Aug 12 16:51:12 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:51:12.732362 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.715121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38724)
0812 16:51:12.715671 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.715703 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:12.715881 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:12.717001 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:12.717008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:12.717546 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:12.717723 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.717734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.719438 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.719448 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.720931 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.720957 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.722167 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.722173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.722482 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.723328 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:12.723352 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:12.726088 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:12.729360 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.729366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.729370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.729964 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.730371 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.730376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.730380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.730519 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.731329 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:12.731334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:12.731491 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 16:51:12.731620 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 16:51:12.731950 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 16:51:12.732451 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.715420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.28.243.61:37921)
0812 16:51:12.716279 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:12.716288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:12.716330 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:51:12.716408 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:12.716418 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:12.716714 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:12.716894 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:12.717931 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.719226 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.721188 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.721956 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.722083 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.726359 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:12.726375 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:12.726392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:12.726432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:12.729217 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.730124 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.730132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.730134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.730265 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.730641 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.730644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.730646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.731183 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:12.731427 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:12.731801 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 16:51:12.732080 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":673,"thread_start_us":434,"threads_started":1}
I20260812 16:51:12.736384 3886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:51:12.738006 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.737360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57236)
0812 16:51:12.737601 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.737787 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 16:51:12.738415 3793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:12.740523 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.740015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57246)
0812 16:51:12.740234 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.740369 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 16:51:12.740887 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab.
Entry for principal kudu/127.28.243.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab.
I20260812 16:51:12.764022 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:37691
--webserver_interface=127.28.243.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab
--principal=kudu/127.28.243.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:43935,127.28.243.61:37921,127.28.243.60:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:51:12.766297 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } attempt: 5
Aug 12 16:51:12 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:51:12.783301 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.767282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.28.243.61:37921)
0812 16:51:12.767444 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:12.767448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:12.767470 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:51:12.767533 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:12.767537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:12.767617 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:12.767711 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:12.768717 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.769454 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.771036 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.771526 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.771632 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.775309 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:12.775317 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:12.775321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:12.775365 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:12.779732 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.780735 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.780741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.780743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.780900 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:12.781356 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:12.781360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:12.781363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:12.781695 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:12.781910 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:12.782782 (+ 872us) server_negotiation.cc:299] Negotiation successful
0812 16:51:12.783071 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 16:51:12.783316 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.767044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38728)
0812 16:51:12.767276 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.767299 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:12.767439 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:12.767766 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:12.767770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:12.768265 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:12.768497 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.768510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.769617 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.769622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.770757 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:12.770773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:12.771686 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:12.771692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:12.771904 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:12.772542 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:12.772565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:12.775050 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:12.780082 (+ 5032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.780088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.780090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.780569 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.781013 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:12.781018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:12.781021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:12.781215 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:12.781881 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:12.781887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:12.782397 (+ 510us) client_negotiation.cc:769] Sending connection context
0812 16:51:12.782612 (+ 215us) client_negotiation.cc:240] Negotiation successful
0812 16:51:12.782952 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 16:51:12.784435 3758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:51:12.785527 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.785108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57248)
0812 16:51:12.785295 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.785376 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 16:51:12.786491 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.786161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57264)
0812 16:51:12.786296 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.786390 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 16:51:12.786708 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:12.790462 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 1
I20260812 16:51:12.791901 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.791319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57266)
0812 16:51:12.791549 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.791655 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 16:51:12.793031 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.792589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57278)
0812 16:51:12.792750 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.792831 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 16:51:12.793294 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:12.837977 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 1
I20260812 16:51:12.839601 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.838761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57294)
0812 16:51:12.839162 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.839355 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260812 16:51:12.841089 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.840480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57308)
0812 16:51:12.840645 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.840901 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 16:51:12.841416 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:12.861559 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 2
I20260812 16:51:12.862829 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.862360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57314)
0812 16:51:12.862560 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.862660 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 16:51:12.863881 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.863528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57318)
0812 16:51:12.863685 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.863759 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 16:51:12.864110 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:12.909667 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 2
I20260812 16:51:12.910902 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.910373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57324)
0812 16:51:12.910628 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.910719 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 16:51:12.912496 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.912001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57332)
0812 16:51:12.912187 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.912271 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 16:51:12.912794 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:12.955358 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 3
I20260812 16:51:12.956701 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.956153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57348)
0812 16:51:12.956419 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.956525 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 16:51:12.958540 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:12.957816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57356)
0812 16:51:12.958136 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:12.958243 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260812 16:51:12.959105 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:13.019101 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 3
I20260812 16:51:13.020783 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.020042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57358)
0812 16:51:13.020400 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.020528 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260812 16:51:13.022017 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.021573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57362)
0812 16:51:13.021787 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.021870 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 16:51:13.022271 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
I20260812 16:51:13.102381 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 4
I20260812 16:51:13.103678 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.103170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57368)
0812 16:51:13.103416 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.103509 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 16:51:13.104677 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.104352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57382)
0812 16:51:13.104497 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.104568 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 16:51:13.104903 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
W20260812 16:51:13.160542 3901 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:51:13.160957 3901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:13.161062 3901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:13.161188 3901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:13.170269 3901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:13.170463 3901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:13.170565 3901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:13.170652 3901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:13.170758 3901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:13.184216 3901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:43935,127.28.243.61:37921,127.28.243.60:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab
--principal=kudu/127.28.243.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:13.186223 3901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:13.187837 3901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 16:51:13.199548 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 4
I20260812 16:51:13.201018 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.200532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57384)
0812 16:51:13.200730 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.200827 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 16:51:13.202498 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.201973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57398)
0812 16:51:13.202219 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.202316 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260812 16:51:13.202907 3758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:37691: Network error: Client connection negotiation failed: client connection to 127.28.243.60:37691: connect: Connection refused (error 111)
W20260812 16:51:13.203495 3909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:13.203886 3908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:13.203527 3911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:13.205650 3901 server_base.cc:1034] running on GCE node
I20260812 16:51:13.206529 3901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:13.208364 3901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:13.209722 3901 hybrid_clock.cc:648] HybridClock initialized: now 1786553473209605 us; error 77 us; skew 500 ppm
I20260812 16:51:13.210230 3901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:13.214534 3901 init.cc:377] Logged in from keytab as kudu/127.28.243.60@KRBTEST.COM (short username kudu)
I20260812 16:51:13.216521 3901 webserver.cc:460] Webserver started at http://127.28.243.60:34469/ using document root <none> and password file <none>
I20260812 16:51:13.217320 3901 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:13.217430 3901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:13.217808 3901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:13.221354 3901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data/instance:
uuid: "4ada3a4180164eb98211baafb7fc138d"
format_stamp: "Formatted at 2026-08-12 16:51:13 on dist-test-slave-1vmg"
server_key: "d3e35f30d3506d22a89ebf5a20c46a2b"
server_key_iv: "3d6bc97c3efa3713654603de66113351"
server_key_version: "encryptionkey@0"
I20260812 16:51:13.222468 3901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal/instance:
uuid: "4ada3a4180164eb98211baafb7fc138d"
format_stamp: "Formatted at 2026-08-12 16:51:13 on dist-test-slave-1vmg"
server_key: "d3e35f30d3506d22a89ebf5a20c46a2b"
server_key_iv: "3d6bc97c3efa3713654603de66113351"
server_key_version: "encryptionkey@0"
I20260812 16:51:13.229321 3901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260812 16:51:13.233774 3924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:13.235400 3901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:51:13.235589 3901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal
uuid: "4ada3a4180164eb98211baafb7fc138d"
format_stamp: "Formatted at 2026-08-12 16:51:13 on dist-test-slave-1vmg"
server_key: "d3e35f30d3506d22a89ebf5a20c46a2b"
server_key_iv: "3d6bc97c3efa3713654603de66113351"
server_key_version: "encryptionkey@0"
I20260812 16:51:13.235843 3901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:13.256546 3901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:13.261251 3901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:13.261555 3901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:13.282303 3901 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.60:37691
I20260812 16:51:13.282321 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.60:37691 every 8 connection(s)
I20260812 16:51:13.284912 3901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
I20260812 16:51:13.288923 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 3901
I20260812 16:51:13.289165 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786553056142401-29644-0/minicluster-data/master-2/wal/instance
I20260812 16:51:13.289593 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:13.289628 29644 external_mini_cluster.cc:1428] Setting key f9c9751af97a470882b495700aee4001
I20260812 16:51:13.300895 4007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:51:13.316310 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.291739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.28.243.60:37691)
0812 16:51:13.292631 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.292652 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.292712 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:51:13.294826 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.294861 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.295505 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.296063 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.297894 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.300069 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.301504 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.302571 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.302763 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.307873 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.307885 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.307904 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.307952 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.311899 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.312812 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.312819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.312827 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.313035 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.313444 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.313447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.313449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.314018 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.314324 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.314803 (+ 479us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.315284 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":356,"threads_started":1}
I20260812 16:51:13.321125 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.304144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:43935 (local address 127.0.0.1:56084)
0812 16:51:13.304937 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.304981 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:13.305142 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:13.305531 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:13.305538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:13.306168 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:13.306431 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.306446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.307587 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.307590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.309159 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.309176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.310034 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.310042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.310246 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.310815 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:13.310843 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:13.313680 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:13.317902 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.317911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.317917 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.318372 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.318827 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.318832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.318835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.319033 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.319578 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:13.319586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:13.319914 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 16:51:13.320132 (+ 218us) client_negotiation.cc:240] Negotiation successful
0812 16:51:13.320600 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":448,"threads_started":1}
I20260812 16:51:13.321333 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.304070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.28.243.62:43935)
0812 16:51:13.304593 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.304599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.305073 (+ 474us) server_negotiation.cc:407] Connection header received
0812 16:51:13.305274 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.305277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.305345 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.305440 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.306595 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.307424 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.309340 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.309827 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.309945 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.313877 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.313882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.313885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.313922 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.317749 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.318532 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.318538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.318541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.318716 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.319150 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.319153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.319156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.319448 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.319606 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.320321 (+ 715us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.320595 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":250,"threads_started":1}
I20260812 16:51:13.327153 4007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:51:13.340207 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.319590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.28.243.62:43935)
0812 16:51:13.321729 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.321734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.321755 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:51:13.321815 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.321821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.321954 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.322097 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.323388 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.324871 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.326732 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.327480 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.328251 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.332767 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.332773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.332777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.332819 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.336535 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.337633 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.337639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.337643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.337809 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.338360 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.338364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.338367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.338668 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.338966 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.339549 (+ 583us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.339829 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":5772,"threads_started":1}
I20260812 16:51:13.346747 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.328003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:37921 (local address 127.0.0.1:38732)
0812 16:51:13.328219 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.328239 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:13.328352 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:13.330673 (+ 2321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:13.330679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:13.331238 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:13.331463 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.331475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.332744 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.332750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.334445 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.334486 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.335845 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.335856 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.336130 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.336912 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:13.336939 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:13.339484 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:13.344236 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.344242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.344246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.344682 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.345114 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.345120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.345124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.345279 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.345893 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:13.345897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:13.346078 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 16:51:13.346213 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 16:51:13.346470 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 16:51:13.347083 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.329789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.28.243.61:37921)
0812 16:51:13.330320 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.330326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.330344 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:51:13.330402 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.330405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.330470 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.330583 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.331710 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.332584 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.334742 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.335455 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.335689 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.339795 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.339803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.339807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.339851 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.344023 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.344827 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.344831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.344833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.344992 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.345395 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.345399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.345401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.345711 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.345899 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.346344 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.346596 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":307,"threads_started":1}
I20260812 16:51:13.348042 4007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:51:13.363418 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.342810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.28.243.61:37921)
0812 16:51:13.343665 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.343671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.343700 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:51:13.343775 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.343780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.343862 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.343991 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.345709 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.346850 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.348548 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.349177 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.349291 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.355993 (+ 6702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.356000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.356004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.356051 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.359977 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.361125 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.361131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.361134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.361330 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.361779 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.361784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.361788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.362188 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.362415 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.362834 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.363100 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":345,"threads_started":1}
I20260812 16:51:13.364786 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.348756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57418)
0812 16:51:13.348939 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.348959 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:13.349067 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:13.349355 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:13.349359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:13.349780 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:13.349955 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.349967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.351329 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.351333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.352201 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.352213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.353371 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.353382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.353663 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.354652 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:13.354682 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:13.357831 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:13.361510 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.361514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.361517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.361993 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.362455 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.362459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.362461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.362595 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.363737 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:13.363741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:13.363901 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 16:51:13.364066 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 16:51:13.364380 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 16:51:13.364859 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.348836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.28.243.60:37691)
0812 16:51:13.348983 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.348987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.349002 (+ 15us) server_negotiation.cc:407] Connection header received
0812 16:51:13.349167 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.349170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.349229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.349319 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.350134 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.351193 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.352388 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.353008 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.353134 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.358247 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.358254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.358259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.358302 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.361331 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.362152 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.362157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.362159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.362318 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.362712 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.362718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.362722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.363506 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.363685 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.364199 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.364566 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":53}
I20260812 16:51:13.369701 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:51:13.370784 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: Bootstrap starting.
I20260812 16:51:13.373101 3707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "800b0de8e9b743eb9512b813154f8c15"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56098
I20260812 16:51:13.376916 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:13.378767 4007 log.cc:824] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:13.383085 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: No bootstrap required, opened a new log
I20260812 16:51:13.390748 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.391199 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ada3a4180164eb98211baafb7fc138d, State: Initialized, Role: FOLLOWER
I20260812 16:51:13.391696 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.392743 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.393157 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.394260 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:13.404112 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 5
W20260812 16:51:13.406179 4048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:51:13.406489 4048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:51:13.403986 3901 master_runner.cc:186] Error getting master registration for 127.28.243.62:43935: OK, instance_id { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" instance_seqno: 1786553472032398 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:51:13.415781 3901 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:51:13.427142 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.405616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.28.243.60:37691)
0812 16:51:13.405842 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.405849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.405868 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:51:13.405927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.405932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.406016 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.406130 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.407363 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.408646 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.410508 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.411638 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.412081 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.417734 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.417742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.417747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.417795 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.422867 (+ 5072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.423957 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.423963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.423967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.424147 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.424920 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.424926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.424930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.425394 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.425620 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.426412 (+ 792us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.426711 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:51:13.427174 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.404811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57424)
0812 16:51:13.405200 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.405224 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:13.405348 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:13.406374 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:13.406379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:13.406949 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:13.407176 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.407188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.408836 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.408842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.410303 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.410324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.412015 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.412023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.412297 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.413189 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:13.413216 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:13.417469 (+ 4253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:13.423117 (+ 5648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.423125 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.423130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.423792 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.424396 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.424405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.424411 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.424728 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.425593 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:13.425600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:13.425844 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 16:51:13.426285 (+ 441us) client_negotiation.cc:240] Negotiation successful
0812 16:51:13.426688 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260812 16:51:13.435246 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: Bootstrap starting.
I20260812 16:51:13.440683 3886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:13.442266 3886 log.cc:824] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:13.446774 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: No bootstrap required, opened a new log
I20260812 16:51:13.460927 3886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.461606 3886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96bbfd666bd242739c32c9e45c335e87, State: Initialized, Role: FOLLOWER
I20260812 16:51:13.462384 3886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.463845 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.464367 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.466181 3886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:13.480175 3758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } attempt: 5
I20260812 16:51:13.475148 3768 master_runner.cc:186] Error getting master registration for 127.28.243.62:43935: OK, instance_id { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" instance_seqno: 1786553472032398 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:51:13.489423 3768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 16:51:13.492851 4069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:51:13.493074 4069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 16:51:13 dist-test-slave-1vmg krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:51:13.511675 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.481448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:37691 (local address 127.0.0.1:57434)
0812 16:51:13.481938 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:13.481959 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:13.482094 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:13.483215 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:13.483221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:13.483776 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:13.483988 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.484002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.485554 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.485560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.487035 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:13.487064 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.488432 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.488438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:13.488666 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.489675 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:13.489701 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:13.495205 (+ 5504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:13.504660 (+ 9455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.504667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.504672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.506001 (+ 1329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.507387 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:13.507394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:13.507399 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:13.507807 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:13.509128 (+ 1321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:13.509136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:13.509451 (+ 315us) client_negotiation.cc:769] Sending connection context
0812 16:51:13.509725 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 16:51:13.510244 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346}
W20260812 16:51:13.514240 3931 tablet.cc:2369] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:51:13.515365 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.482474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.28.243.60:37691)
0812 16:51:13.482675 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.482681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.482701 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:51:13.482770 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.482775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.482854 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.483005 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.484194 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.485369 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.487270 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.488025 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.488165 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.495621 (+ 7456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.495629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.495635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.495703 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.504286 (+ 8583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.506587 (+ 2301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.506606 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.506610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.507029 (+ 419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.508073 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.508079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.508083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.508792 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.511606 (+ 2814us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.512644 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.513622 (+ 978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 16:51:13.524124 3758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15: Bootstrap starting.
I20260812 16:51:13.528934 3758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:13.530568 3758 log.cc:824] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:13.538873 3758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15: No bootstrap required, opened a new log
I20260812 16:51:13.553702 3758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.554530 3758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800b0de8e9b743eb9512b813154f8c15, State: Initialized, Role: FOLLOWER
I20260812 16:51:13.555548 3758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.557562 4074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.558115 4074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.561810 3707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 16:51:13.562059 3707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:13.563031 3758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:13.565841 3707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.568779 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800b0de8e9b743eb9512b813154f8c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "96bbfd666bd242739c32c9e45c335e87"
I20260812 16:51:13.569355 3841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:13.574529 3841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800b0de8e9b743eb9512b813154f8c15 in term 1.
I20260812 16:51:13.575699 3657 leader_election.cc:302] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800b0de8e9b743eb9512b813154f8c15, 96bbfd666bd242739c32c9e45c335e87; no voters:
I20260812 16:51:13.576427 4074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:51:13.577795 3966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800b0de8e9b743eb9512b813154f8c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ada3a4180164eb98211baafb7fc138d"
I20260812 16:51:13.578718 3966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:13.583555 3966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800b0de8e9b743eb9512b813154f8c15 in term 1.
I20260812 16:51:13.584472 3707 leader_election.cc:288] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [CANDIDATE]: Term 1 election: Requested vote from peers 96bbfd666bd242739c32c9e45c335e87 (127.28.243.61:37921), 4ada3a4180164eb98211baafb7fc138d (127.28.243.60:37691)
I20260812 16:51:13.587234 4074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [term 1 LEADER]: Becoming Leader. State: Replica: 800b0de8e9b743eb9512b813154f8c15, State: Running, Role: LEADER
I20260812 16:51:13.588032 4074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } }
I20260812 16:51:13.601511 4078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800b0de8e9b743eb9512b813154f8c15. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.602094 4078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:13.619882 4085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:51:13.643136 4085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260812 16:51:13.646061 3801 tablet.cc:2369] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:51:13.648700 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.605510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.28.243.61:37921)
0812 16:51:13.605714 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.605721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.613226 (+ 7505us) server_negotiation.cc:407] Connection header received
0812 16:51:13.616387 (+ 3161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.616394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.616494 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.616585 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.624687 (+ 8102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.626123 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.629819 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.630619 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.630754 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.640260 (+ 9506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.640269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.640275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.640325 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.644522 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.645589 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.645595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.645599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.645767 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.646260 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.646265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.646268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.646671 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.646974 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.648030 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.648328 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 16:51:13.650877 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.607872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.28.243.62:43935)
0812 16:51:13.608070 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.608076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.619605 (+ 11529us) server_negotiation.cc:407] Connection header received
0812 16:51:13.619939 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.619946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.620038 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.620125 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.623172 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.624319 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.626579 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.627436 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.627559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.643162 (+ 15603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.643171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.643176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.643253 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.647102 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.647995 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.648001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.648004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.648165 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.648626 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.648631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.648634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.649073 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.649333 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.650251 (+ 918us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.650490 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:51:13.662113 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:13.613856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.28.243.60:37691)
0812 16:51:13.614065 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:13.614071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:13.620336 (+ 6265us) server_negotiation.cc:407] Connection header received
0812 16:51:13.621508 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:13.621514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:13.621880 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:13.622049 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:13.628297 (+ 6248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.629366 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.636237 (+ 6871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:13.636955 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:13.637093 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:13.653702 (+ 16609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:13.653710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:13.653716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:13.653765 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:13.657861 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.658952 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.658958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.658962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.659130 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:13.659640 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:13.659646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:13.659650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:13.660096 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:13.660355 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:13.661010 (+ 655us) server_negotiation.cc:299] Negotiation successful
0812 16:51:13.661569 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:51:13.692523 3966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [term 1 FOLLOWER]: Refusing update from remote peer 800b0de8e9b743eb9512b813154f8c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:51:13.694999 3841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [term 1 FOLLOWER]: Refusing update from remote peer 800b0de8e9b743eb9512b813154f8c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:51:13.695919 4074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:51:13.702713 4074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:51:13.737928 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800b0de8e9b743eb9512b813154f8c15. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.738368 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.752288 4108 mvcc.cc:205] Tried to move back new op lower bound from 7317723028218220544 to 7317723027824242688. Current Snapshot: MvccSnapshot[applied={T|T < 7317723028218220544}]
I20260812 16:51:13.754603 4078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.754992 4078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:13.759399 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.759697 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.762405 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800b0de8e9b743eb9512b813154f8c15. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.762792 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.765895 4074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.766270 4074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:13.767299 4085 catalog_manager.cc:1347] Generated new cluster ID: 5a636f67d6694ba9b12d957a70fd9b62
I20260812 16:51:13.767402 4085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:51:13.770257 4110 mvcc.cc:205] Tried to move back new op lower bound from 7317723028218220544 to 7317723027824242688. Current Snapshot: MvccSnapshot[applied={T|T < 7317723028218220544}]
I20260812 16:51:13.784682 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "800b0de8e9b743eb9512b813154f8c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b0de8e9b743eb9512b813154f8c15" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 43935 } } peers { permanent_uuid: "96bbfd666bd242739c32c9e45c335e87" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 37921 } } peers { permanent_uuid: "4ada3a4180164eb98211baafb7fc138d" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 37691 } } }
I20260812 16:51:13.784974 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:13.827678 4085 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:51:13.829687 4085 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:51:13.853561 4085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 800b0de8e9b743eb9512b813154f8c15: Generated new TSK 0
I20260812 16:51:13.854816 4085 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:51:14.021672 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.003335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.28.243.61:37921)
0812 16:51:14.003534 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.003551 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.016081 (+ 12530us) server_negotiation.cc:407] Connection header received
0812 16:51:14.019806 (+ 3725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.019813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.019919 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.020012 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.021376 (+ 1364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38770: BlockingRecv error: recv got EOF from 127.0.0.1:38770 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 16:51:14.025695 4012 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.011857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.28.243.62:43935)
0812 16:51:14.012122 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.012130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.022823 (+ 10693us) server_negotiation.cc:407] Connection header received
0812 16:51:14.023926 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.023932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.024029 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.024115 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.025404 (+ 1289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56126: BlockingRecv error: recv got EOF from 127.0.0.1:56126 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 16:51:14.028859 4011 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.017149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.28.243.60:37691)
0812 16:51:14.017365 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.017372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.023604 (+ 6232us) server_negotiation.cc:407] Connection header received
0812 16:51:14.027103 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.027110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.027218 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.027304 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.028574 (+ 1270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57440: BlockingRecv error: recv got EOF from 127.0.0.1:57440 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 16:51:14.075256 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.065289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.28.243.61:37921)
0812 16:51:14.065476 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.065482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.073722 (+ 8240us) server_negotiation.cc:407] Connection header received
0812 16:51:14.073958 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.073962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.074093 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.074187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.075024 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38784: BlockingRecv error: recv got EOF from 127.0.0.1:38784 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 16:51:14.082615 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.080457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.28.243.61:37921)
0812 16:51:14.080681 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.080698 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.081232 (+ 534us) server_negotiation.cc:407] Connection header received
0812 16:51:14.081419 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.081423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.081501 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.081593 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.082316 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38792: BlockingRecv error: recv got EOF from 127.0.0.1:38792 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 16:51:14.089908 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.087729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.28.243.61:37921)
0812 16:51:14.087942 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.087948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.088455 (+ 507us) server_negotiation.cc:407] Connection header received
0812 16:51:14.088660 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.088664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.088758 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.088845 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.089628 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38806: BlockingRecv error: recv got EOF from 127.0.0.1:38806 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 16:51:14.100988 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.098779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.28.243.61:37921)
0812 16:51:14.099015 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.099023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.099541 (+ 518us) server_negotiation.cc:407] Connection header received
0812 16:51:14.099772 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.099778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.099858 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.099950 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.100720 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38810: BlockingRecv error: recv got EOF from 127.0.0.1:38810 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 16:51:14.114817 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.112639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.28.243.61:37921)
0812 16:51:14.112858 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.112865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.113398 (+ 533us) server_negotiation.cc:407] Connection header received
0812 16:51:14.113601 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.113607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.113701 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.113793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.114584 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38822: BlockingRecv error: recv got EOF from 127.0.0.1:38822 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 16:51:14.136880 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.134689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.28.243.61:37921)
0812 16:51:14.134951 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.134957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.135468 (+ 511us) server_negotiation.cc:407] Connection header received
0812 16:51:14.135684 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.135688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.135759 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.135842 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.136572 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38828: BlockingRecv error: recv got EOF from 127.0.0.1:38828 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 16:51:14.175434 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.173064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.28.243.61:37921)
0812 16:51:14.173280 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.173288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.173853 (+ 565us) server_negotiation.cc:407] Connection header received
0812 16:51:14.174059 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.174063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.174148 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.174237 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.175177 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38844: BlockingRecv error: recv got EOF from 127.0.0.1:38844 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 16:51:14.245358 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.243020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.28.243.61:37921)
0812 16:51:14.243219 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.243225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.243557 (+ 332us) server_negotiation.cc:407] Connection header received
0812 16:51:14.243750 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.243755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.243883 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.244028 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.245105 (+ 1077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38848: BlockingRecv error: recv got EOF from 127.0.0.1:38848 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 16:51:14.378894 4019 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:14.376823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.28.243.61:37921)
0812 16:51:14.377067 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.377074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.377318 (+ 244us) server_negotiation.cc:407] Connection header received
0812 16:51:14.377526 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.377532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.377606 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.377695 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:14.378648 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38860: BlockingRecv error: recv got EOF from 127.0.0.1:38860 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:51:14.408936 4048 catalog_manager.cc:1259] Loaded cluster ID: 5a636f67d6694ba9b12d957a70fd9b62
I20260812 16:51:14.409082 4048 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: loading cluster ID for follower catalog manager: success
I20260812 16:51:14.411973 4048 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: acquiring CA information for follower catalog manager: success
I20260812 16:51:14.414215 4048 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ada3a4180164eb98211baafb7fc138d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:51:14.496343 4069 catalog_manager.cc:1259] Loaded cluster ID: 5a636f67d6694ba9b12d957a70fd9b62
I20260812 16:51:14.496455 4069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: loading cluster ID for follower catalog manager: success
I20260812 16:51:14.499359 4069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: acquiring CA information for follower catalog manager: success
I20260812 16:51:14.501438 4069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96bbfd666bd242739c32c9e45c335e87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:51:14.647584 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:14.640557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.28.243.61:37921)
0812 16:51:14.640880 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.640919 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.641395 (+ 476us) server_negotiation.cc:407] Connection header received
0812 16:51:14.641568 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.641572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.641771 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.641889 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:14.642416 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.643320 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.644841 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.645299 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.645422 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:14.646234 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:14.646816 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:14.647007 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:14.647213 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 16:51:14.647310 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:51:14.663787 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:14.655885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.28.243.60:37691)
0812 16:51:14.656482 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.656489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.656690 (+ 201us) server_negotiation.cc:407] Connection header received
0812 16:51:14.657093 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.657102 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.657376 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.657635 (+ 259us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:14.658334 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.659347 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.661220 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.661672 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.661804 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:14.662394 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:14.663019 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:14.663166 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:14.663279 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 16:51:14.663375 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260812 16:51:14.684338 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:14.672759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.28.243.61:37921)
0812 16:51:14.673114 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.673123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.674710 (+ 1587us) server_negotiation.cc:407] Connection header received
0812 16:51:14.676286 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.676292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.676393 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.676456 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:14.677677 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.678834 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.681918 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.682625 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.682744 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:14.683468 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:14.683635 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:14.683692 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:14.683928 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 16:51:14.684032 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:51:14.685856 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:14.672272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.28.243.62:43935)
0812 16:51:14.674171 (+ 1899us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.674179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.675593 (+ 1414us) server_negotiation.cc:407] Connection header received
0812 16:51:14.676174 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.676185 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.676480 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.676745 (+ 265us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:14.677824 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.679277 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.682224 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.682975 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.683111 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:14.683947 (+ 836us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:14.684648 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:14.684775 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:14.685391 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 16:51:14.685508 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1722,"thread_start_us":1448,"threads_started":1}
I20260812 16:51:14.691365 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:14.675248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.28.243.60:37691)
0812 16:51:14.679439 (+ 4191us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:14.679446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:14.685152 (+ 5706us) server_negotiation.cc:407] Connection header received
0812 16:51:14.685417 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:14.685422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:14.685523 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:14.685588 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:14.686170 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.687316 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.689534 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:14.689955 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:14.690072 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:14.690500 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:14.690621 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:14.690698 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:14.690853 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 16:51:14.691068 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3717}
I20260812 16:51:14.702272 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3635
I20260812 16:51:14.703832 3752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:14.834098 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3635
W20260812 16:51:14.876358 3796 connection.cc:535] server connection from 127.0.0.1:38728 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 16:51:14.877329 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3768
I20260812 16:51:14.879153 3874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:15.006798 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3768
I20260812 16:51:15.048960 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 3901
I20260812 16:51:15.050313 4001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:15.203176 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 3901
2026-08-12T16:51:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5478 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:51:15 dist-test-slave-1vmg krb5kdc[4248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:51:15 dist-test-slave-1vmg krb5kdc[4248](info): set up 2 sockets
Aug 12 16:51:15 dist-test-slave-1vmg krb5kdc[4248](info): commencing operation
krb5kdc: starting...
W20260812 16:51:17.158634 29644 mini_kdc.cc:121] Time spent starting KDC: real 1.874s user 0.000s sys 0.111s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/test-user.keytab.
Aug 12 16:51:17 dist-test-slave-1vmg krb5kdc[4248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
Entry for principal kudu/127.28.243.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab.
I20260812 16:51:17.587760 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46743
--webserver_interface=127.28.243.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46743,127.28.243.61:41357,127.28.243.60:35279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:51:17.994035 4264 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:51:17.994413 4264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:17.994490 4264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:17.994575 4264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:18.002485 4264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:18.002609 4264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:18.002684 4264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:18.002739 4264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:18.002806 4264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:18.014158 4264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46743,127.28.243.61:41357,127.28.243.60:35279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.62.keytab
--principal=kudu/127.28.243.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.62:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:18.015642 4264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:18.017057 4264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:18.031741 4280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:18.031740 4278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:18.031647 4277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:18.033016 4264 server_base.cc:1034] running on GCE node
I20260812 16:51:18.033910 4264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:18.035800 4264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:18.037113 4264 hybrid_clock.cc:648] HybridClock initialized: now 1786553478037036 us; error 52 us; skew 500 ppm
I20260812 16:51:18.037528 4264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:18 dist-test-slave-1vmg krb5kdc[4248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:18.041519 4264 init.cc:377] Logged in from keytab as kudu/127.28.243.62@KRBTEST.COM (short username kudu)
I20260812 16:51:18.043572 4264 webserver.cc:460] Webserver started at http://127.28.243.62:33099/ using document root <none> and password file <none>
I20260812 16:51:18.044399 4264 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:18.044526 4264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:18.044888 4264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:18.048693 4264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data/instance:
uuid: "a2cecc2d05a744e4804d3ae959d33e17"
format_stamp: "Formatted at 2026-08-12 16:51:18 on dist-test-slave-1vmg"
server_key: "6393b48afd64416a0ede6a1a1624c10c"
server_key_iv: "25a6127d40c8073c4566c0f1a60ded8d"
server_key_version: "encryptionkey@0"
I20260812 16:51:18.049638 4264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal/instance:
uuid: "a2cecc2d05a744e4804d3ae959d33e17"
format_stamp: "Formatted at 2026-08-12 16:51:18 on dist-test-slave-1vmg"
server_key: "6393b48afd64416a0ede6a1a1624c10c"
server_key_iv: "25a6127d40c8073c4566c0f1a60ded8d"
server_key_version: "encryptionkey@0"
I20260812 16:51:18.056604 4264 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 16:51:18.061153 4292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:18.063068 4264 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:51:18.063292 4264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal
uuid: "a2cecc2d05a744e4804d3ae959d33e17"
format_stamp: "Formatted at 2026-08-12 16:51:18 on dist-test-slave-1vmg"
server_key: "6393b48afd64416a0ede6a1a1624c10c"
server_key_iv: "25a6127d40c8073c4566c0f1a60ded8d"
server_key_version: "encryptionkey@0"
I20260812 16:51:18.063539 4264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:18.078157 4264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:18.083869 4264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:18.084192 4264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:18.105708 4264 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.62:46743
I20260812 16:51:18.105748 4377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.62:46743 every 8 connection(s)
I20260812 16:51:18.107344 4264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/data/info.pb
I20260812 16:51:18.110592 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4264
I20260812 16:51:18.110816 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-0/wal/instance
I20260812 16:51:18.111289 29644 external_mini_cluster.cc:1428] Setting key 49b99ea0d74e6b4024f440303c0eeb26
I20260812 16:51:18.111727 4380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:18.122517 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:18 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.61@KRBTEST.COM; defaulting to no policy
I20260812 16:51:18.151008 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.127447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46743 (local address 127.0.0.1:57728)
0812 16:51:18.128483 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.128543 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:18.130798 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:18.131831 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:18.131838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:18.132722 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:18.133092 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.133122 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.134762 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.134766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.136114 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.136130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.137569 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.137580 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.137788 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.138527 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:18.138571 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:18.141595 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:18.146339 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.146355 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.146372 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.147113 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.147654 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.147660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.147664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.147899 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.148876 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:18.148883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:18.149107 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 16:51:18.149356 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 16:51:18.149829 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":431,"threads_started":1}
I20260812 16:51:18.151077 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.127446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.28.243.62:46743)
0812 16:51:18.129415 (+ 1969us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:18.129424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:18.129481 (+ 57us) server_negotiation.cc:407] Connection header received
0812 16:51:18.131295 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:18.131302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:18.131575 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:18.131784 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:18.133333 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.134584 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.136295 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.137197 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.137412 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.142087 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:18.142103 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:18.142127 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:18.142194 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:18.146133 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.147308 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.147317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.147319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.147516 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.148090 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.148095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.148097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.148733 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:18.148926 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:18.149591 (+ 665us) server_negotiation.cc:299] Negotiation successful
0812 16:51:18.149999 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":1726,"spinlock_wait_cycles":22784,"thread_start_us":1486,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.61@KRBTEST.COM" created.
I20260812 16:51:18.154955 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:51:18.156178 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.155652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60510)
0812 16:51:18.155851 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.156011 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 16:51:18.156490 4294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
I20260812 16:51:18.158672 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.158177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60522)
0812 16:51:18.158377 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.158474 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 16:51:18.158901 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:41357: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab.
Entry for principal kudu/127.28.243.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab.
I20260812 16:51:18.199824 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:41357
--webserver_interface=127.28.243.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab
--principal=kudu/127.28.243.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46743,127.28.243.61:41357,127.28.243.60:35279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:51:18.208379 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } attempt: 1
I20260812 16:51:18.209745 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.209227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60534)
0812 16:51:18.209438 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.209569 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 16:51:18.211045 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.210553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60546)
0812 16:51:18.210716 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.210808 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 16:51:18.211267 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:41357: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
I20260812 16:51:18.279554 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } attempt: 2
I20260812 16:51:18.280918 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.280412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60556)
0812 16:51:18.280660 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.280762 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 16:51:18.282276 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.281808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60568)
0812 16:51:18.282000 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.282118 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260812 16:51:18.282461 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:41357: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
I20260812 16:51:18.373740 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } attempt: 3
I20260812 16:51:18.375016 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.374514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60570)
0812 16:51:18.374734 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.374827 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 16:51:18.376058 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.375675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60582)
0812 16:51:18.375836 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.375913 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 16:51:18.376255 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:41357: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
I20260812 16:51:18.519593 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } attempt: 4
I20260812 16:51:18.521492 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.520670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60588)
0812 16:51:18.521154 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.521290 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272}
I20260812 16:51:18.523109 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.522513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60600)
0812 16:51:18.522719 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.522849 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 16:51:18.523351 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.61:41357: Network error: Client connection negotiation failed: client connection to 127.28.243.61:41357: connect: Connection refused (error 111)
W20260812 16:51:18.592262 4390 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:51:18.592650 4390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:18.592762 4390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:18.592883 4390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:51:18.600821 4390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:18.600946 4390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:18.601030 4390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:18.601100 4390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:18.601192 4390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:18.612756 4390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46743,127.28.243.61:41357,127.28.243.60:35279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.61.keytab
--principal=kudu/127.28.243.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.61:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:18.614192 4390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:18.615800 4390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:18.629783 4401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:18.629760 4404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:18.629740 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:18.630617 4390 server_base.cc:1034] running on GCE node
I20260812 16:51:18.631587 4390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:18.633139 4390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:18.634472 4390 hybrid_clock.cc:648] HybridClock initialized: now 1786553478634364 us; error 70 us; skew 500 ppm
I20260812 16:51:18.634869 4390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:18 dist-test-slave-1vmg krb5kdc[4248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:18.638630 4390 init.cc:377] Logged in from keytab as kudu/127.28.243.61@KRBTEST.COM (short username kudu)
I20260812 16:51:18.640650 4390 webserver.cc:460] Webserver started at http://127.28.243.61:39673/ using document root <none> and password file <none>
I20260812 16:51:18.641451 4390 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:18.641556 4390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:18.641903 4390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:18.646098 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data/instance:
uuid: "95252765106049ad87fbef5be93a78e6"
format_stamp: "Formatted at 2026-08-12 16:51:18 on dist-test-slave-1vmg"
server_key: "74e3787490a73d7908a90823441645dd"
server_key_iv: "74319157595e0778622b82a395676111"
server_key_version: "encryptionkey@0"
I20260812 16:51:18.647157 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal/instance:
uuid: "95252765106049ad87fbef5be93a78e6"
format_stamp: "Formatted at 2026-08-12 16:51:18 on dist-test-slave-1vmg"
server_key: "74e3787490a73d7908a90823441645dd"
server_key_iv: "74319157595e0778622b82a395676111"
server_key_version: "encryptionkey@0"
I20260812 16:51:18.653179 4390 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:51:18.657599 4419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:18.659246 4390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:51:18.659458 4390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal
uuid: "95252765106049ad87fbef5be93a78e6"
format_stamp: "Formatted at 2026-08-12 16:51:18 on dist-test-slave-1vmg"
server_key: "74e3787490a73d7908a90823441645dd"
server_key_iv: "74319157595e0778622b82a395676111"
server_key_version: "encryptionkey@0"
I20260812 16:51:18.659703 4390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:18.672191 4390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:18.677120 4390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:18.677409 4390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:18.698999 4390 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.61:41357
I20260812 16:51:18.699069 4493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.61:41357 every 8 connection(s)
I20260812 16:51:18.700611 4390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/data/info.pb
I20260812 16:51:18.702632 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4390
I20260812 16:51:18.702850 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-1/wal/instance
I20260812 16:51:18.703503 29644 external_mini_cluster.cc:1428] Setting key 5ec9525eba8d1753228322096e3c6ff7
I20260812 16:51:18.705937 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:18.716516 4496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:18 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.60@KRBTEST.COM; defaulting to no policy
I20260812 16:51:18.743376 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.721140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.28.243.62:46743)
0812 16:51:18.721754 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:18.721761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:18.722853 (+ 1092us) server_negotiation.cc:407] Connection header received
0812 16:51:18.725075 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:18.725081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:18.725185 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:18.725293 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:18.727209 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.728008 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.729669 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.730107 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.730239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.735381 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:18.735388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:18.735392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:18.735441 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:18.739228 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.740379 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.740384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.740387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.740536 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.741040 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.741044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.741046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.741405 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:18.741582 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:18.742561 (+ 979us) server_negotiation.cc:299] Negotiation successful
0812 16:51:18.742875 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":300,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.60@KRBTEST.COM" created.
I20260812 16:51:18.744911 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.721695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46743 (local address 127.0.0.1:57732)
0812 16:51:18.722656 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.722708 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:18.724714 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:18.725601 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:18.725612 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:18.726253 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:18.726903 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.726968 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.728204 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.728209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.729491 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.729506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.730387 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.730396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.730660 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.731418 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:18.731446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:18.735033 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:18.739503 (+ 4470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.739517 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.739539 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.740204 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.740662 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.740667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.740669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.740904 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.741561 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:18.741572 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:18.742104 (+ 532us) client_negotiation.cc:769] Sending connection context
0812 16:51:18.742434 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 16:51:18.743190 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":348,"threads_started":1}
I20260812 16:51:18.747529 4496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:18 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:51:18.767231 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.748382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60608)
0812 16:51:18.748601 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.748621 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:18.748734 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:18.749956 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:18.749960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:18.750440 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:18.750578 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.750586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.753037 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.753042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.754654 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.754685 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.756136 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.756142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.756332 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.757059 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:18.757079 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:18.760110 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:18.764382 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.764388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.764393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.764865 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.765297 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.765303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.765307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.765524 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.766263 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:18.766268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:18.766432 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 16:51:18.766579 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 16:51:18.766899 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 16:51:18.767694 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.748721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.28.243.61:41357)
0812 16:51:18.749469 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:18.749475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:18.749512 (+ 37us) server_negotiation.cc:407] Connection header received
0812 16:51:18.749562 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:18.749568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:18.749778 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:18.749946 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:18.750747 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.752564 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.755178 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.755950 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.756103 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.760412 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:18.760440 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:18.760464 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:18.760509 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:18.764211 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.764997 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.765004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.765006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.765141 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.765698 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.765702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.765705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.766110 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:18.766297 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:18.766819 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 16:51:18.767112 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":418,"threads_started":1}
I20260812 16:51:18.770517 4496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:51:18.771931 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.771242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57608)
0812 16:51:18.771446 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.771579 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 16:51:18.772351 4421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:18.774763 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.774315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57612)
0812 16:51:18.774511 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.774614 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 16:51:18.775262 4496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab.
Entry for principal kudu/127.28.243.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab.
I20260812 16:51:18.801090 29644 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
/tmp/dist-test-taskWMmoHF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:35279
--webserver_interface=127.28.243.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab
--principal=kudu/127.28.243.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.62:46743,127.28.243.61:41357,127.28.243.60:35279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:51:18.822674 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } attempt: 5
I20260812 16:51:18.824688 4496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 1
I20260812 16:51:18.825816 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.825316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57616)
0812 16:51:18.825494 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.825629 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 16:51:18.827116 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.826653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57620)
0812 16:51:18.826840 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.826966 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 16:51:18.827363 4496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Aug 12 16:51:18 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:51:18.842228 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.823554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60612)
0812 16:51:18.823750 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.823772 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:18.823888 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:18.824315 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:18.824320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:18.824816 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:18.825067 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.825081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.826438 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.826444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.827628 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:18.827643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.828582 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.828587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:18.828759 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.829355 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:18.829373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:18.831786 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:18.837971 (+ 6185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.837978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.837982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.838804 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.840026 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:18.840056 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:18.840061 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:18.840311 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:18.841266 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:18.841272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:18.841478 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 16:51:18.841645 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 16:51:18.841928 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 16:51:18.842449 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.823817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.28.243.61:41357)
0812 16:51:18.824014 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:18.824019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:18.824035 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:51:18.824089 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:18.824092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:18.824171 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:18.824257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:18.825218 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.826254 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.827776 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:18.828439 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:18.828571 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:18.832278 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:18.832289 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:18.832295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:18.832373 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:18.837566 (+ 5193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.839173 (+ 1607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.839182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.839185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.839687 (+ 502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:18.840578 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:18.840585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:18.840589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:18.841089 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:18.841291 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:18.841812 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 16:51:18.842091 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:51:18.843700 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:51:18.845052 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.844491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57624)
0812 16:51:18.844736 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.844846 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 16:51:18.846063 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.845676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57636)
0812 16:51:18.845826 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.845935 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 16:51:18.846297 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:18.895666 4496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 2
I20260812 16:51:18.897550 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.896787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57650)
0812 16:51:18.897132 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.897282 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 16:51:18.897564 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 1
I20260812 16:51:18.899190 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.898290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57666)
0812 16:51:18.898640 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.898895 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 16:51:18.899397 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.898649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57682)
0812 16:51:18.899082 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.899200 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260812 16:51:18.899708 4496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:18.900677 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.900231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57684)
0812 16:51:18.900398 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.900503 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:51:18.900924 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:18.969344 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 2
I20260812 16:51:18.970521 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.970019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57694)
0812 16:51:18.970246 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.970353 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 16:51:18.971472 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.971145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57700)
0812 16:51:18.971293 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.971366 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 16:51:18.971915 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:18.991021 4496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 3
I20260812 16:51:18.992937 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.992226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57710)
0812 16:51:18.992495 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.992613 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 16:51:18.995241 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:18.994642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57714)
0812 16:51:18.995004 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:18.995105 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260812 16:51:18.995505 4496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:19.078255 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 3
I20260812 16:51:19.080180 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.079376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57728)
0812 16:51:19.079717 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.079894 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 16:51:19.081624 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.081076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57738)
0812 16:51:19.081336 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.081442 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260812 16:51:19.081899 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
I20260812 16:51:19.138746 4496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 4
I20260812 16:51:19.140228 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.139690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57750)
0812 16:51:19.139955 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.140051 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 16:51:19.142055 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.141546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57752)
0812 16:51:19.141741 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.141833 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 16:51:19.142452 4496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
W20260812 16:51:19.257114 4507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:51:19.257480 4507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:51:19.257587 4507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:51:19.257696 4507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 16:51:19.259137 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 4
I20260812 16:51:19.261097 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.260322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57762)
0812 16:51:19.260715 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.260870 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 16:51:19.262702 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.262126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57770)
0812 16:51:19.262348 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.262474 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 16:51:19.262987 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.60:35279: Network error: Client connection negotiation failed: client connection to 127.28.243.60:35279: connect: Connection refused (error 111)
W20260812 16:51:19.267976 4507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:51:19.268077 4507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:51:19.268159 4507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:51:19.268227 4507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:51:19.268321 4507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:51:19.282648 4507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.62:46743,127.28.243.61:41357,127.28.243.60:35279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/krb5kdc/kudu_127.28.243.60.keytab
--principal=kudu/127.28.243.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.60:35279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:51:19.284817 4507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:51:19.286458 4507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:51:19.300855 4516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:19.301322 4520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:51:19.302563 4515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:51:19.302551 4507 server_base.cc:1034] running on GCE node
I20260812 16:51:19.303826 4507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:51:19.305644 4507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:51:19.306901 4507 hybrid_clock.cc:648] HybridClock initialized: now 1786553479306832 us; error 44 us; skew 500 ppm
I20260812 16:51:19.307349 4507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553479, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:51:19.311082 4507 init.cc:377] Logged in from keytab as kudu/127.28.243.60@KRBTEST.COM (short username kudu)
I20260812 16:51:19.313436 4507 webserver.cc:460] Webserver started at http://127.28.243.60:36511/ using document root <none> and password file <none>
I20260812 16:51:19.314608 4507 fs_manager.cc:359] Metadata directory not provided
I20260812 16:51:19.314751 4507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:51:19.315270 4507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:51:19.320035 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data/instance:
uuid: "e447c4a2965f4cde87734497d3fd208a"
format_stamp: "Formatted at 2026-08-12 16:51:19 on dist-test-slave-1vmg"
server_key: "ca16109100de6ab0305f79af1ff5ec1a"
server_key_iv: "98c046f94abb672385d42c05d9f453e4"
server_key_version: "encryptionkey@0"
I20260812 16:51:19.321511 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal/instance:
uuid: "e447c4a2965f4cde87734497d3fd208a"
format_stamp: "Formatted at 2026-08-12 16:51:19 on dist-test-slave-1vmg"
server_key: "ca16109100de6ab0305f79af1ff5ec1a"
server_key_iv: "98c046f94abb672385d42c05d9f453e4"
server_key_version: "encryptionkey@0"
I20260812 16:51:19.328090 4507 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 16:51:19.333500 4533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:51:19.335314 4507 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:51:19.335539 4507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data,/tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal
uuid: "e447c4a2965f4cde87734497d3fd208a"
format_stamp: "Formatted at 2026-08-12 16:51:19 on dist-test-slave-1vmg"
server_key: "ca16109100de6ab0305f79af1ff5ec1a"
server_key_iv: "98c046f94abb672385d42c05d9f453e4"
server_key_version: "encryptionkey@0"
I20260812 16:51:19.335803 4507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:51:19.354820 4507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:51:19.361558 4507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:51:19.361892 4507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:51:19.384727 4507 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.60:35279
I20260812 16:51:19.384747 4617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.60:35279 every 8 connection(s)
I20260812 16:51:19.386266 4507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/data/info.pb
I20260812 16:51:19.388743 29644 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu as pid 4507
I20260812 16:51:19.388958 29644 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWMmoHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786553056142401-29644-0/minicluster-data/master-2/wal/instance
I20260812 16:51:19.389518 29644 external_mini_cluster.cc:1428] Setting key e03c3abb2af4409a1a75538535dfc630
I20260812 16:51:19.390960 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:51:19.403177 4618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553479, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
I20260812 16:51:19.422003 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.391605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.28.243.60:35279)
0812 16:51:19.392451 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.392463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.392507 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:51:19.394328 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.394354 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.394838 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.395379 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.398564 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.400451 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.402887 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.403927 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.404151 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.409279 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.409291 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.409306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.409368 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.414590 (+ 5222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.415991 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.416004 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.416018 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.416563 (+ 545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.417456 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.417462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.417465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.418619 (+ 1154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.419242 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.419971 (+ 729us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.420477 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":390,"threads_started":1}
I20260812 16:51:19.427553 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.406874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.62:46743 (local address 127.0.0.1:57734)
0812 16:51:19.408289 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.408349 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:19.408552 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:19.409107 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:19.409115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:19.410259 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:19.410692 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.410709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.412139 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.412148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.414589 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.414603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.416098 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.416106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.416562 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.417491 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:19.417521 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:19.420454 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:19.424248 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.424259 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.424266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.424943 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.425406 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.425411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.425414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.425572 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.426332 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:19.426341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:19.426600 (+ 259us) client_negotiation.cc:769] Sending connection context
0812 16:51:19.426847 (+ 247us) client_negotiation.cc:240] Negotiation successful
0812 16:51:19.427227 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":871,"threads_started":1}
I20260812 16:51:19.428129 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.406911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.28.243.62:46743)
0812 16:51:19.407753 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.407761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.408477 (+ 716us) server_negotiation.cc:407] Connection header received
0812 16:51:19.408705 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.408710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.408820 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.408954 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.410953 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.411866 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.414831 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.415722 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.415891 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.420734 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.420742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.420746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.420794 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.424081 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.425104 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.425110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.425114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.425268 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.425698 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.425704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.425707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.426159 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.426339 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.427082 (+ 743us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.427334 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":470,"threads_started":1}
I20260812 16:51:19.429715 4618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.62@KRBTEST.COM
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553479, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
I20260812 16:51:19.441745 4496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 5
I20260812 16:51:19.447122 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.425847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.28.243.62:46743)
0812 16:51:19.426491 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.426496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.426521 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:51:19.426568 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.426572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.426641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.426751 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.428757 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.429868 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.431577 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.432231 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.432375 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.438602 (+ 6227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.438611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.438616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.438678 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.442777 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.444163 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.444171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.444175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.444386 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.444913 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.444918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.444921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.445298 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.445699 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.446134 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.446434 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":304,"threads_started":1}
I20260812 16:51:19.448958 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.430482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.61:41357 (local address 127.0.0.1:60618)
0812 16:51:19.430761 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.430781 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:19.430890 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:19.431550 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:19.431554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:19.432088 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:19.432299 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.432312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.433830 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.433837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.435025 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.435048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.436169 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.436177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.436596 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.437484 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:19.437505 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:19.440400 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:19.445542 (+ 5142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.445548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.445553 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.446187 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.446753 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.446759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.446763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.446980 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.447833 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:19.447838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:19.448040 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 16:51:19.448198 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 16:51:19.448575 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260812 16:51:19.449306 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.430673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.28.243.61:41357)
0812 16:51:19.431222 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.431227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.431246 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:51:19.431304 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.431307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.431383 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.431491 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.432513 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.433546 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.435249 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.435821 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.435942 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.440773 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.440782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.440786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.440839 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.445298 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.446377 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.446383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.446386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.446600 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.447152 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.447163 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.447167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.447628 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.447978 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.448415 (+ 437us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.448697 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":450,"thread_start_us":337,"threads_started":1}
I20260812 16:51:19.450879 4618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.61@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:51:19.462734 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.442660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57788)
0812 16:51:19.443100 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.443122 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:19.443280 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:19.443937 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:19.443944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:19.444609 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:19.444907 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.444923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.446458 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.446466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.447909 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.447927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.449112 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.449119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.449863 (+ 744us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.450528 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:19.450557 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:19.453567 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:19.459624 (+ 6057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.459631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.459634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.460224 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.460704 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.460707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.460710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.460852 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.461618 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:19.461624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:19.461802 (+ 178us) client_negotiation.cc:769] Sending connection context
0812 16:51:19.461945 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 16:51:19.462297 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260812 16:51:19.463936 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.442879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.28.243.60:35279)
0812 16:51:19.443163 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.443169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.443191 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:51:19.443453 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.443458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.443559 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.443679 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.445165 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.446291 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.448109 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.448750 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.448881 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.453857 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.453864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.453868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.453916 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.459299 (+ 5383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.460380 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.460385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.460387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.460571 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.460978 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.460982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.460984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.461467 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.463069 (+ 1602us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.463254 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.463524 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":126}
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.61@KRBTEST.COM
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553479, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.60@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:51:19.470736 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: Bootstrap starting.
I20260812 16:51:19.476799 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.451965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57804)
0812 16:51:19.452294 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.452314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:19.452417 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:19.457385 (+ 4968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:19.457400 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:19.458157 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:19.458417 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.458436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.461173 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.461180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.462529 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.462546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.463822 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.463828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.464023 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.464724 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:19.464753 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:19.467621 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:19.472112 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.472118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.472122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.472744 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.473175 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.473180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.473184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.473323 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.474618 (+ 1295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:19.474625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:19.475093 (+ 468us) client_negotiation.cc:769] Sending connection context
0812 16:51:19.475329 (+ 236us) client_negotiation.cc:240] Negotiation successful
0812 16:51:19.476010 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":25}
I20260812 16:51:19.476959 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.451869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.28.243.60:35279)
0812 16:51:19.456217 (+ 4348us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.456224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.456260 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:51:19.456456 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.456463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.456589 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.456754 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.458765 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.460913 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.462733 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.463633 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.463791 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.467848 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.467854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.467858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.467900 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.471814 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.472884 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.472890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.472894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.473050 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.473424 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.473430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.473433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.474041 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.474625 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.475493 (+ 868us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.476315 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":499,"server-negotiator.queue_time_us":3938,"thread_start_us":3429,"threads_started":1}
I20260812 16:51:19.477499 4496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:19.478361 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.450294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.28.243.61:41357)
0812 16:51:19.450606 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.450613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.451432 (+ 819us) server_negotiation.cc:407] Connection header received
0812 16:51:19.451489 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.451492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.451556 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.451651 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.454731 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.456538 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.459948 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.460836 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.461159 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.468371 (+ 7212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.468379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.468383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.468430 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.472379 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.473874 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.473880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.473883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.474493 (+ 610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.475036 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.475041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.475043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.475410 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.475765 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.477624 (+ 1859us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.477907 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260812 16:51:19.479496 4496 log.cc:824] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:19.483994 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: No bootstrap required, opened a new log
I20260812 16:51:19.484210 4618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: Bootstrap starting.
I20260812 16:51:19.485239 29644 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:51:19.488560 4618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:19.489802 4618 log.cc:824] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:19.488840 4338 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a2cecc2d05a744e4804d3ae959d33e17"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57738
I20260812 16:51:19.494647 4496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.495210 4496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95252765106049ad87fbef5be93a78e6, State: Initialized, Role: FOLLOWER
I20260812 16:51:19.495976 4496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.498049 4618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: No bootstrap required, opened a new log
I20260812 16:51:19.498278 4633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.499003 4633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.499967 4496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:19.509989 4618 raft_consensus.cc:348] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.510525 4618 raft_consensus.cc:729] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e447c4a2965f4cde87734497d3fd208a, State: Initialized, Role: FOLLOWER
W20260812 16:51:19.510951 4650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:51:19.511107 4650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:51:19.511125 4618 consensus_queue.cc:260] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.508430 4390 master_runner.cc:186] Error getting master registration for 127.28.243.62:46743: OK, instance_id { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" instance_seqno: 1786553478099312 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:51:19.522609 4390 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 16:51:19.523723 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.524233 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.525411 4618 sys_catalog.cc:564] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:51:19.538964 4675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:51:19.539440 4675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:51:19.540210 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } attempt: 5
I20260812 16:51:19.535231 4507 master_runner.cc:186] Error getting master registration for 127.28.243.62:46743: OK, instance_id { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" instance_seqno: 1786553478099312 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:51:19.549129 4507 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 16:51:19 dist-test-slave-1vmg krb5kdc[4248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553478, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.62@KRBTEST.COM for kudu/127.28.243.60@KRBTEST.COM
I20260812 16:51:19.562940 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.541712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.60:35279 (local address 127.0.0.1:57814)
0812 16:51:19.542018 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 16:51:19.542041 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:51:19.542164 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:51:19.542741 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:51:19.542746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:51:19.543317 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:51:19.543543 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.543556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.545176 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.545182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.546696 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:51:19.546714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.548107 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.548115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:51:19.548341 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.549170 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:51:19.549210 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:51:19.552716 (+ 3506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:51:19.559298 (+ 6582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.559305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.559309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.560092 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.560705 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:51:19.560710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:51:19.560713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:51:19.560928 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:51:19.561749 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:51:19.561755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:51:19.561950 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 16:51:19.562133 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 16:51:19.562475 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260812 16:51:19.564229 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.541762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.28.243.60:35279)
0812 16:51:19.542417 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.542423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.542445 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:51:19.542504 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.542509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.542592 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.542684 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.543754 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.544941 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.546945 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.547654 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.547802 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.553632 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.553643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.553649 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.553721 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.558730 (+ 5009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.560304 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.560311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.560315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.560535 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.561075 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.561080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.561083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.561472 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.561655 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.563417 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.563713 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502}
I20260812 16:51:19.571048 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17: Bootstrap starting.
I20260812 16:51:19.576853 4380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17: Neither blocks nor log segments found. Creating new log.
I20260812 16:51:19.579372 4380 log.cc:824] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17: Log is configured to *not* fsync() on all Append() calls
I20260812 16:51:19.584625 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17: No bootstrap required, opened a new log
I20260812 16:51:19.594662 4380 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.595664 4380 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cecc2d05a744e4804d3ae959d33e17, State: Initialized, Role: FOLLOWER
I20260812 16:51:19.596472 4380 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.598356 4681 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.599435 4380 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:51:19.599864 4338 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 16:51:19.600564 4338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:19.604648 4338 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.607435 4460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2cecc2d05a744e4804d3ae959d33e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95252765106049ad87fbef5be93a78e6"
I20260812 16:51:19.607990 4460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:19.611999 4460 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cecc2d05a744e4804d3ae959d33e17 in term 1.
W20260812 16:51:19.614584 4543 tablet.cc:2369] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:51:19.616396 4294 leader_election.cc:302] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95252765106049ad87fbef5be93a78e6, a2cecc2d05a744e4804d3ae959d33e17; no voters:
I20260812 16:51:19.620463 4581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2cecc2d05a744e4804d3ae959d33e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e447c4a2965f4cde87734497d3fd208a"
I20260812 16:51:19.621060 4581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:51:19.624881 4581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cecc2d05a744e4804d3ae959d33e17 in term 1.
I20260812 16:51:19.626255 4338 leader_election.cc:288] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [CANDIDATE]: Term 1 election: Requested vote from peers 95252765106049ad87fbef5be93a78e6 (127.28.243.61:41357), e447c4a2965f4cde87734497d3fd208a (127.28.243.60:35279)
I20260812 16:51:19.629302 4681 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.629885 4681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:51:19.630465 4681 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [term 1 LEADER]: Becoming Leader. State: Replica: a2cecc2d05a744e4804d3ae959d33e17, State: Running, Role: LEADER
I20260812 16:51:19.631448 4681 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } }
I20260812 16:51:19.653095 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2cecc2d05a744e4804d3ae959d33e17. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.654222 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:19.656760 4706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 16:51:19.681628 4426 tablet.cc:2369] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:51:19.692245 4706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:51:19.717532 4720 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 16:51:19.724946 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.666400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.28.243.62:46743)
0812 16:51:19.666643 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.666650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.683371 (+ 16721us) server_negotiation.cc:407] Connection header received
0812 16:51:19.683707 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.683713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.683820 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.683943 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.686479 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.687653 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.691034 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.692007 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.692123 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.706153 (+ 14030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.706161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.706167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.706214 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.710285 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.711649 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.711660 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.711664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.711905 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.713765 (+ 1860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.713774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.713779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.714548 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.715507 (+ 959us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.724229 (+ 8722us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.724579 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:51:19.739655 4460 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [term 1 FOLLOWER]: Refusing update from remote peer a2cecc2d05a744e4804d3ae959d33e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:51:19.741780 4681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:51:19.744567 4581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [term 1 FOLLOWER]: Refusing update from remote peer a2cecc2d05a744e4804d3ae959d33e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:51:19.746260 4681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:51:19.770888 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.667427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.28.243.60:35279)
0812 16:51:19.667641 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.667648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.684162 (+ 16514us) server_negotiation.cc:407] Connection header received
0812 16:51:19.684450 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.684457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.684542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.684611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.695582 (+ 10971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.696883 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.704691 (+ 7808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.705383 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.705495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.722505 (+ 17010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.722514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.722519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.722589 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.730539 (+ 7950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.763817 (+ 33278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.763826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.763831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.764134 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.767173 (+ 3039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.767180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.767184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.767635 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.768135 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.770145 (+ 2010us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.770453 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:51:19.795858 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:19.685362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.28.243.61:41357)
0812 16:51:19.685541 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:19.685547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:19.689259 (+ 3712us) server_negotiation.cc:407] Connection header received
0812 16:51:19.692777 (+ 3518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:19.692784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:19.692883 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:19.692981 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:19.712531 (+ 19550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.714256 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.723682 (+ 9426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:19.724612 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:19.724837 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:19.777382 (+ 52545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:51:19.777392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:51:19.777398 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:51:19.777464 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:51:19.781580 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.790808 (+ 9228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.790815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.790819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.791126 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:51:19.791869 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:51:19.791885 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:51:19.791888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:51:19.792361 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:51:19.792681 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:19.795073 (+ 2392us) server_negotiation.cc:299] Negotiation successful
0812 16:51:19.795397 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:51:19.804538 4633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2cecc2d05a744e4804d3ae959d33e17. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.804916 4633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.815367 4725 mvcc.cc:205] Tried to move back new op lower bound from 7317723052969136128 to 7317723052629729280. Current Snapshot: MvccSnapshot[applied={T|T < 7317723052969136128}]
I20260812 16:51:19.821696 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2cecc2d05a744e4804d3ae959d33e17. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.822088 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.826326 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.826647 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:19.828222 4633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.828502 4633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.848778 4722 mvcc.cc:205] Tried to move back new op lower bound from 7317723052969136128 to 7317723052629729280. Current Snapshot: MvccSnapshot[applied={T|T < 7317723052969136128}]
I20260812 16:51:19.850648 4681 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.851184 4681 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17 [sys.catalog]: This master's current role is: LEADER
I20260812 16:51:19.862670 4706 catalog_manager.cc:1347] Generated new cluster ID: dfe606d3e0d74a16a35296c12a90a077
I20260812 16:51:19.862824 4706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:51:19.870512 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2cecc2d05a744e4804d3ae959d33e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cecc2d05a744e4804d3ae959d33e17" member_type: VOTER last_known_addr { host: "127.28.243.62" port: 46743 } } peers { permanent_uuid: "95252765106049ad87fbef5be93a78e6" member_type: VOTER last_known_addr { host: "127.28.243.61" port: 41357 } } peers { permanent_uuid: "e447c4a2965f4cde87734497d3fd208a" member_type: VOTER last_known_addr { host: "127.28.243.60" port: 35279 } } }
I20260812 16:51:19.870880 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:51:19.897334 4706 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:51:19.903527 4706 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:51:19.927585 4706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2cecc2d05a744e4804d3ae959d33e17: Generated new TSK 0
I20260812 16:51:19.929180 4706 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:51:20.127301 4627 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.089141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.28.243.62:46743)
0812 16:51:20.089354 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.089360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.124917 (+ 35557us) server_negotiation.cc:407] Connection header received
0812 16:51:20.125474 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.125480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.125575 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.125649 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.127009 (+ 1360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57750: BlockingRecv error: recv got EOF from 127.0.0.1:57750 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 16:51:20.133309 4629 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.096033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.28.243.60:35279)
0812 16:51:20.096324 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.096334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.129711 (+ 33377us) server_negotiation.cc:407] Connection header received
0812 16:51:20.130045 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.130052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.130145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.130213 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.133016 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57834: BlockingRecv error: recv got EOF from 127.0.0.1:57834 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
W20260812 16:51:20.137444 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.135308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.28.243.61:41357)
0812 16:51:20.135516 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.135522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.135655 (+ 133us) server_negotiation.cc:407] Connection header received
0812 16:51:20.135957 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.135962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.136050 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.136128 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.137192 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60644: BlockingRecv error: recv got EOF from 127.0.0.1:60644 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 16:51:20.159310 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.156732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.28.243.61:41357)
0812 16:51:20.156956 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.156963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.157557 (+ 594us) server_negotiation.cc:407] Connection header received
0812 16:51:20.157824 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.157830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.157971 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.158081 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.159048 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60660: BlockingRecv error: recv got EOF from 127.0.0.1:60660 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 16:51:20.171111 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.168576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.28.243.61:41357)
0812 16:51:20.168767 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.168773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.169398 (+ 625us) server_negotiation.cc:407] Connection header received
0812 16:51:20.169654 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.169660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.169770 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.169880 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.170851 (+ 971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60676: BlockingRecv error: recv got EOF from 127.0.0.1:60676 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 16:51:20.180124 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.177872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.28.243.61:41357)
0812 16:51:20.178047 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.178051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.178407 (+ 356us) server_negotiation.cc:407] Connection header received
0812 16:51:20.178622 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.178625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.178700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.178789 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.179781 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60682: BlockingRecv error: recv got EOF from 127.0.0.1:60682 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 16:51:20.190819 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.188680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.28.243.61:41357)
0812 16:51:20.188918 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.188923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.189200 (+ 277us) server_negotiation.cc:407] Connection header received
0812 16:51:20.189405 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.189409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.189484 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.189590 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.190569 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60696: BlockingRecv error: recv got EOF from 127.0.0.1:60696 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260812 16:51:20.204824 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.202783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.28.243.61:41357)
0812 16:51:20.202974 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.202979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.203356 (+ 377us) server_negotiation.cc:407] Connection header received
0812 16:51:20.203553 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.203557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.203650 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.203742 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.204579 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 16:51:20.227432 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.224952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.28.243.61:41357)
0812 16:51:20.225191 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.225198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.225832 (+ 634us) server_negotiation.cc:407] Connection header received
0812 16:51:20.225947 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.225952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.226074 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.226200 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.227183 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60706: BlockingRecv error: recv got EOF from 127.0.0.1:60706 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 16:51:20.266964 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.263960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.28.243.61:41357)
0812 16:51:20.264228 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.264235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.265087 (+ 852us) server_negotiation.cc:407] Connection header received
0812 16:51:20.265406 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.265412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.265533 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.265644 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.266682 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60718: BlockingRecv error: recv got EOF from 127.0.0.1:60718 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260812 16:51:20.338263 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.336183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.28.243.61:41357)
0812 16:51:20.336359 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.336365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.336769 (+ 404us) server_negotiation.cc:407] Connection header received
0812 16:51:20.336957 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.336963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.337048 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.337136 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.337923 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60728: BlockingRecv error: recv got EOF from 127.0.0.1:60728 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 16:51:20.472837 4628 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:51:20.470826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.28.243.61:41357)
0812 16:51:20.471034 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.471039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.471429 (+ 390us) server_negotiation.cc:407] Connection header received
0812 16:51:20.471570 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.471574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.471654 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.471754 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:51:20.472548 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60734: BlockingRecv error: recv got EOF from 127.0.0.1:60734 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:51:20.513260 4650 catalog_manager.cc:1259] Loaded cluster ID: dfe606d3e0d74a16a35296c12a90a077
I20260812 16:51:20.513377 4650 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: loading cluster ID for follower catalog manager: success
I20260812 16:51:20.516312 4650 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: acquiring CA information for follower catalog manager: success
I20260812 16:51:20.518355 4650 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95252765106049ad87fbef5be93a78e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:51:20.541749 4675 catalog_manager.cc:1259] Loaded cluster ID: dfe606d3e0d74a16a35296c12a90a077
I20260812 16:51:20.541863 4675 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: loading cluster ID for follower catalog manager: success
I20260812 16:51:20.544510 4675 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: acquiring CA information for follower catalog manager: success
I20260812 16:51:20.546561 4675 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e447c4a2965f4cde87734497d3fd208a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:51:20.740635 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:20.733856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.28.243.61:41357)
0812 16:51:20.734099 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.734106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.734265 (+ 159us) server_negotiation.cc:407] Connection header received
0812 16:51:20.734614 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.734619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.734808 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.734960 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:20.735573 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.736297 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.737895 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.738495 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.738615 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:20.739278 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:20.739886 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:20.740054 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:20.740176 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 16:51:20.740271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 16:51:20.755299 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:20.747642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.28.243.60:35279)
0812 16:51:20.748142 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.748147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.748167 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:51:20.748279 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.748284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.748442 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.748597 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:20.749394 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.750262 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.752393 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.752877 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.753001 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:20.753676 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:20.754268 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:20.754394 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:20.754664 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 16:51:20.754829 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":272,"threads_started":1}
I20260812 16:51:20.770179 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:20.762298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.28.243.60:35279)
0812 16:51:20.762480 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.762487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.763458 (+ 971us) server_negotiation.cc:407] Connection header received
0812 16:51:20.763731 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.763737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.763826 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.763885 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:20.764750 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.765895 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.768190 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.768713 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.768857 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:20.769442 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:20.769555 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:20.769638 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:20.769777 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 16:51:20.769866 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:51:20.770599 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:20.762133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.28.243.62:46743)
0812 16:51:20.762582 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.762586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.763102 (+ 516us) server_negotiation.cc:407] Connection header received
0812 16:51:20.763388 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.763395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.763637 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.763849 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:20.764781 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.765895 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.767844 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.768526 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.768672 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:20.769308 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:20.769796 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:20.769889 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:20.770217 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 16:51:20.770304 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
I20260812 16:51:20.771792 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:51:20.763517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.28.243.61:41357)
0812 16:51:20.763721 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 16:51:20.763728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:51:20.764264 (+ 536us) server_negotiation.cc:407] Connection header received
0812 16:51:20.764461 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:51:20.764466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:51:20.764557 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:51:20.764635 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:51:20.766576 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.767776 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.769618 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:51:20.770256 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:51:20.770382 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:51:20.771001 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:51:20.771131 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:51:20.771206 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:51:20.771436 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 16:51:20.771519 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:51:20.775691 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4264
I20260812 16:51:20.776639 4368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:20.888825 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4264
I20260812 16:51:20.926546 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4390
I20260812 16:51:20.927702 4488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:21.041062 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4390
I20260812 16:51:21.071738 29644 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu pid 4507
I20260812 16:51:21.073338 4612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:51:21.186484 29644 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWMmoHF/build/asan/bin/kudu with pid 4507
2026-08-12T16:51:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5954 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11433 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (424958 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST